Oct 30 01:27:59.766817 ip-10-0-142-110 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:28:00.151062 ip-10-0-142-110 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:00.151062 ip-10-0-142-110 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:28:00.151062 ip-10-0-142-110 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:00.151062 ip-10-0-142-110 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:00.155106 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155034 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:00.155106 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155104 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:00.155106 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155108 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155111 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155114 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155117 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155120 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155122 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155125 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155128 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155130 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155133 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155136 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155138 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155143 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155148 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155151 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155155 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155158 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155161 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155163 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:00.155201 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155166 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155168 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155171 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155174 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155176 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155179 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155191 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155194 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155197 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155199 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155202 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155205 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155208 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155211 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155215 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155218 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155221 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155223 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155226 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:00.155670 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155228 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155231 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155234 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155236 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155239 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155242 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155244 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155247 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155250 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155253 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155255 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155258 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155261 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155265 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155267 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155272 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155276 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155279 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155282 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:00.156116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155284 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155287 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155290 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155293 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155297 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155299 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155302 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155305 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155307 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155310 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155313 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155316 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155318 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155320 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155323 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155326 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155328 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155331 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155333 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155336 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:00.156568 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155339 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155341 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155344 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155346 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155349 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155353 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155355 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155358 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155361 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155363 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155366 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155368 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155371 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155374 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155377 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155379 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155382 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155384 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155387 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:00.157135 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155389 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155392 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155394 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155397 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155401 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155404 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155406 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.155409 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156177 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156184 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156187 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156190 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156192 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156197 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156201 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156204 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156207 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156210 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156213 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:00.157641 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156216 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156219 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156222 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156225 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156229 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156233 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156236 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156238 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156241 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156244 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156247 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156249 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156252 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156255 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156257 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156260 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156263 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156265 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156268 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:00.158086 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156271 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156274 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156276 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156279 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156281 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156284 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156287 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156289 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156292 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156294 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156297 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156300 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156303 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156305 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156310 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156313 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156315 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156318 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156320 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:00.158561 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156323 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156326 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156328 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156331 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156334 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156337 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156339 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156342 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156344 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156347 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156349 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156352 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156355 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156357 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156360 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156362 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156365 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156367 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156370 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:00.159013 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156375 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156377 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156380 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156383 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156385 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156388 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156391 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156394 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156396 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156400 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156402 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156405 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156408 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156410 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156413 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156415 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156418 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156420 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156423 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156426 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:00.159465 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156428 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156430 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156433 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156435 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156438 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156441 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156444 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156447 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156449 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156452 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156454 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156457 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156460 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156463 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156466 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156468 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156471 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.156474 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157052 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157059 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:00.159966 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157063 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157067 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157077 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157080 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157083 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157085 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157088 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157091 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157093 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157096 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157099 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157101 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157104 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157107 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157110 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157112 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157115 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157118 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157120 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:00.160439 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157123 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157126 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157128 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157130 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157133 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157138 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157141 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157144 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157146 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157151 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157154 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157157 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157160 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157162 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157165 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157174 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157177 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157179 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157182 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157184 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:00.160907 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157187 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157189 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157192 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157195 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157197 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157200 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157202 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157205 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157207 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157210 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157213 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157215 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157218 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157221 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157224 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157228 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157231 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157236 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157239 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157241 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:00.161361 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157244 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157249 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157253 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157256 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157259 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157262 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157264 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157267 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157275 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157278 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157281 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157283 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157286 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157289 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157291 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157294 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157296 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157299 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157301 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157304 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:00.161854 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157307 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157309 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157312 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157314 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157317 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157320 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157322 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157325 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157327 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157332 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157334 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157338 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157342 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157347 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157350 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157353 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157356 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157360 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157363 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:00.162350 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157366 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157368 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157376 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157379 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157382 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.157384 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158316 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158324 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158332 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158337 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158341 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158345 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158349 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158353 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158357 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158360 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158363 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158367 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158370 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158373 2576 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158376 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158379 2576 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158384 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:28:00.162825 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158387 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158391 2576 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158394 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158398 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158401 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158405 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158409 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158412 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158415 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158418 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158422 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158425 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158428 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158432 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158435 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158439 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158442 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158445 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158448 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158451 2576 flags.go:64] FLAG: --enable-server="true" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158454 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158459 2576 flags.go:64] FLAG: --event-burst="100" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158462 2576 flags.go:64] FLAG: --event-qps="50" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158465 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:28:00.163363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158468 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158471 2576 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158475 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158478 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158481 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158484 2576 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158487 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158492 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158495 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158498 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158500 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158504 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158507 2576 flags.go:64] FLAG: --feature-gates="" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158511 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158514 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158517 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158520 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158538 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158542 2576 flags.go:64] FLAG: --help="false" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158545 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158548 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158552 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158555 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158558 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:28:00.163986 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158562 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158565 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158568 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158571 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158575 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158578 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158581 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158584 2576 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158586 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158589 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158592 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158595 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158598 2576 flags.go:64] FLAG: --lock-file="" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158601 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158604 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158608 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158614 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158616 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158619 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158622 2576 flags.go:64] FLAG: --logging-format="text" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158626 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158629 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158632 2576 flags.go:64] FLAG: --manifest-url="" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158635 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158644 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:28:00.164592 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158647 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158651 2576 flags.go:64] FLAG: --max-pods="110" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158654 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158657 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158660 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158663 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158667 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158670 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158673 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158679 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158682 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158685 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158688 2576 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158691 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158694 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158697 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158700 2576 flags.go:64] FLAG: --port="10250" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158703 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158706 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07cc7ab149845010e" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158709 2576 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158712 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158716 2576 flags.go:64] FLAG: --register-node="true" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158719 2576 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158722 2576 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158725 2576 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158728 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:28:00.165223 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158731 2576 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158735 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158738 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158741 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158744 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158747 2576 flags.go:64] FLAG: --runonce="false" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158750 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158753 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158756 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158759 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158762 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158766 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158769 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158772 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158775 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158778 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158781 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158784 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158787 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158790 2576 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158793 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158798 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158801 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158804 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158808 2576 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:28:00.165896 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158810 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158813 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158816 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158821 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158824 2576 flags.go:64] FLAG: --v="2" Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158828 2576 flags.go:64] FLAG: --version="false" Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158832 2576 flags.go:64] FLAG: --vmodule="" Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158836 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.158839 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159016 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159020 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159024 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159026 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159029 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159032 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159035 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159038 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159040 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159043 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159046 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159055 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:00.166468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159058 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159061 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159064 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159066 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159069 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159072 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159075 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159077 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159080 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159082 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159085 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159088 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159090 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159093 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159098 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159100 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159103 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159106 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159108 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:00.166980 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159111 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159114 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159117 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159119 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159122 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159125 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159128 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159130 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159133 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159136 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159138 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159141 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159144 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159152 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159155 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159157 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159160 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159163 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159165 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159168 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:00.167428 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159171 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159173 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159181 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159185 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159187 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159190 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159209 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159215 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159218 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159221 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159224 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159227 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159230 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159232 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159236 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159240 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159244 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159246 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159250 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159253 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:00.168012 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159255 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159258 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159261 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159264 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159266 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159269 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159279 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159281 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159284 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159287 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159289 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159292 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159294 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159297 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159300 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159302 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159305 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159308 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159312 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:00.168498 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159316 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159319 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159322 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159325 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159327 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159330 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159334 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159338 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159341 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159344 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159346 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159349 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159352 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159355 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159357 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:00.168953 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159360 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.159385 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159426 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159430 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159433 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159441 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159444 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159447 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159449 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159452 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159454 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159457 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159460 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159462 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159465 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159467 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:00.169320 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159472 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159475 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159479 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159482 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159484 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159487 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159490 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159492 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159495 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159499 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159503 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159506 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159509 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159511 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159514 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159517 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159519 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159522 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159536 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159539 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:00.169710 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159542 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159544 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159553 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159556 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159558 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159561 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159564 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159566 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159569 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159571 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159574 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159577 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159581 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159583 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159588 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159590 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159593 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159596 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159599 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:00.170195 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159602 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159606 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159610 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159613 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159616 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159619 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159621 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159624 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159635 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159638 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159640 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159643 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159645 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159648 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159651 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159653 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159662 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159665 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159667 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:00.170667 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159670 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159672 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159675 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159678 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159681 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159685 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159689 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159691 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159696 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159699 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159702 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159704 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159707 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159710 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159713 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159715 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159718 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159720 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159723 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:00.171098 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159726 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159729 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159731 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159734 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159737 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159739 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159742 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159745 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159748 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159751 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159753 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159762 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159764 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159767 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.159770 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.167171 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.167186 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.167871 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.167877 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167952 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:00.171687 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167957 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167960 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167963 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167966 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167969 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167972 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167975 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167977 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167980 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167983 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167986 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167988 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167991 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167994 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167997 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.167999 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168008 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168011 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168014 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168017 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:00.172174 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168019 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168022 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168025 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168028 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168031 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168034 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168036 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168039 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168042 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168045 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168048 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168051 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168053 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168056 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168059 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168061 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168064 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168068 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168071 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:00.172659 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168074 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168077 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168080 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168083 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168086 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168088 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168091 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168094 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168096 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168099 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168107 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168111 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168114 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168117 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168119 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168122 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168125 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168128 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168130 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168133 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:00.173102 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168135 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168144 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168147 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168150 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168152 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168155 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168158 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168161 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168164 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168168 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168172 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168175 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168178 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168180 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168183 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168186 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168189 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168191 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168194 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:00.173592 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168197 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168199 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168202 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168205 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168214 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168217 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168219 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168222 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168225 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168227 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168230 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168233 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168235 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168238 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168241 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168243 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168246 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168248 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168251 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:00.174036 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168254 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168256 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168259 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168261 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168264 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168266 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168269 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168272 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.168298 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168513 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168519 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168522 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168539 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168543 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168546 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168548 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168551 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:00.175215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168554 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168564 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168567 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168570 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168573 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168575 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168578 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168581 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168583 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168586 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168588 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168591 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168593 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168596 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168599 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168601 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168604 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168607 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168609 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168612 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:00.175911 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168615 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168617 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168620 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168623 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168625 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168628 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168631 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168633 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168636 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168638 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168641 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168643 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168646 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168649 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168658 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168661 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168664 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168666 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168669 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:00.176485 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168671 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168674 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168676 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168679 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168681 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168684 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168687 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168690 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168692 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168695 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168697 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168700 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168702 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168706 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168711 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168714 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168717 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168720 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168723 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:00.177019 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168725 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168728 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168730 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168733 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168735 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168738 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168741 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168744 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168747 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168755 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168758 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168760 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168763 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168766 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168768 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168771 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168773 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168776 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168779 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:00.177468 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168781 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168784 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168787 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168789 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168792 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168794 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168797 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168799 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168802 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168804 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168807 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168810 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168812 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168815 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168817 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168820 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168824 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168827 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168831 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:00.177929 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168834 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168841 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.168864 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168907 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168917 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168920 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168923 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168926 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168929 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168932 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168934 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168937 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168939 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168942 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168945 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168947 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:00.178380 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168950 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168953 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168956 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168959 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168961 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168964 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168967 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168969 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168972 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168975 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168977 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168980 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168982 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168985 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168988 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168990 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.168994 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169000 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169003 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:00.178784 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169006 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169010 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169019 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169022 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169025 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169028 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169030 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169033 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169035 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169038 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169040 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169044 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169047 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169049 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169052 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169054 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169057 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169060 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169062 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169065 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:00.179259 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169067 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169070 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169072 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169075 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169077 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169080 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169082 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169085 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169088 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169090 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169094 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169097 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169100 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169102 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169105 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169113 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169116 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169119 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169121 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169129 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:00.179752 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169132 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169135 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169137 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169140 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169142 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169145 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169147 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169150 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169152 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169155 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169157 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169160 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169163 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169165 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169168 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169170 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169173 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169176 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169178 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169181 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:00.180215 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169183 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169186 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169191 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169195 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169198 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169200 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169203 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169206 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169214 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169217 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169219 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169222 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169224 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:00.169228 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.169285 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.171576 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.172519 2576 server.go:1014] "Starting client certificate rotation" Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.172648 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:28:00.180695 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.173547 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:28:00.181107 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.176937 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:28:00.181107 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.178074 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:28:00.189924 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.189909 2576 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:28:00.193385 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.193369 2576 log.go:25] "Validated CRI v1 image API" Oct 30 01:28:00.194264 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.194229 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:28:00.198467 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.198451 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ec2340c-3279-43d2-83d0-34fd083e8244:/dev/nvme0n1p4 aa7532c3-5652-44b6-9b48-e764a5cf4ac9:/dev/nvme0n1p3] Oct 30 01:28:00.198514 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.198467 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 01:28:00.204612 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.204483 2576 manager.go:217] Machine: {Timestamp:2026-10-30 01:28:00.202091541 +0000 UTC m=+0.322738875 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101034 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec247cd7daafc93fef2201935a2b7ea3 SystemUUID:ec247cd7-daaf-c93f-ef22-01935a2b7ea3 BootID:e71f8d96-feab-4af0-847e-6e65d4db1120 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:a3:2e:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:a3:2e:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:af:4c:c1:d9:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:28:00.204696 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.204614 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:28:00.204696 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.204689 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:28:00.208790 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.208767 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:28:00.208937 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.208791 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:28:00.208983 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.208940 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:28:00.208983 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.208948 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:28:00.209036 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.208982 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:28:00.209036 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.209000 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:28:00.210001 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.209991 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:28:00.214099 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.214084 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:28:00.214238 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.214230 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:28:00.214991 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.214982 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:28:00.215027 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.214995 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:28:00.215027 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.215008 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:28:00.215027 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.215017 2576 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:28:00.215123 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.215064 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:28:00.215412 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.215400 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:28:00.217988 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.217969 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:28:00.223777 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.223760 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:28:00.224665 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.224650 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:28:00.224734 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.224682 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:28:00.224834 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.224823 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:28:00.224864 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.224840 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:28:00.224864 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.224847 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:28:00.224864 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.224853 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:28:00.224864 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.224859 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:28:00.224864 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.224864 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:28:00.225557 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.225547 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:28:00.225589 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.225557 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:28:00.225589 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.225565 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:28:00.225589 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.225572 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:28:00.225589 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.225580 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:28:00.225589 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.225589 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:28:00.227155 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.227143 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:28:00.227193 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.227158 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:28:00.230912 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.230897 2576 server.go:1263] "Started kubelet" Oct 30 01:28:00.231004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.230975 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:28:00.231063 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.230992 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:28:00.231102 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.231082 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:28:00.231607 ip-10-0-142-110 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:28:00.232676 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.232406 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:28:00.234121 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.233228 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:28:00.238017 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.237997 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 01:28:00.240051 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.240034 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:28:00.240618 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.240600 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:28:00.240906 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.240866 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:00.241320 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241304 2576 factory.go:55] Registering systemd factory Oct 30 01:28:00.241320 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241321 2576 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:28:00.241464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241339 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:28:00.241464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241342 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:28:00.241464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241364 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:28:00.241464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241454 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:28:00.241464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241463 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:28:00.241696 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241508 2576 factory.go:153] Registering CRI-O factory Oct 30 01:28:00.241696 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241517 2576 factory.go:223] Registration of the crio container factory successfully Oct 30 01:28:00.241696 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241585 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:28:00.241696 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241598 2576 factory.go:103] Registering Raw factory Oct 30 01:28:00.241696 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.241608 2576 manager.go:1199] Started watching for new ooms in manager Oct 30 01:28:00.242829 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.242812 2576 manager.go:319] Starting recovery of all containers Oct 30 01:28:00.253787 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.253625 2576 manager.go:324] Recovery completed Oct 30 01:28:00.256355 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.256332 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 01:28:00.258670 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.258648 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 01:28:00.258765 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.258659 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 01:28:00.259689 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.259676 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:00.261374 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.261359 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:00.261433 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.261388 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:00.261433 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.261398 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:00.261932 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.261919 2576 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:28:00.261932 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.261929 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:00.262005 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.261932 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:28:00.262005 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.262001 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:28:00.262603 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.262590 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:00.262647 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.262615 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:00.262647 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.262624 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:00.264170 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.264155 2576 policy_none.go:50] "Start" Oct 30 01:28:00.264221 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.264176 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:28:00.264221 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.264190 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:28:00.266226 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.266215 2576 policy_none.go:44] "Start" Oct 30 01:28:00.268777 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.267683 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-110.ec2.internal.18e32a62bf751377 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-110.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-110.ec2.internal,},FirstTimestamp:2026-10-30 01:28:00.230871927 +0000 UTC m=+0.351519260,LastTimestamp:2026-10-30 01:28:00.230871927 +0000 UTC m=+0.351519260,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-110.ec2.internal,}" Oct 30 01:28:00.278952 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.278930 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 01:28:00.279026 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.278951 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 01:28:00.279026 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.279009 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 01:28:00.280616 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.280598 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:28:00.300019 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.299951 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-110.ec2.internal.18e32a62c1468330 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-110.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-110.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-110.ec2.internal,},FirstTimestamp:2026-10-30 01:28:00.261374768 +0000 UTC m=+0.382022101,LastTimestamp:2026-10-30 01:28:00.261374768 +0000 UTC m=+0.382022101,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-110.ec2.internal,}" Oct 30 01:28:00.305824 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.305808 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnr8d" Oct 30 01:28:00.306108 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.306078 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:28:00.309880 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.309861 2576 manager.go:341] "Starting Device Plugin manager" Oct 30 01:28:00.309954 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.309891 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:28:00.309954 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.309900 2576 server.go:87] "Starting device plugin registration server" Oct 30 01:28:00.310123 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.310111 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:28:00.310170 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.310124 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:28:00.310221 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.310207 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:28:00.310302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.310289 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:28:00.310341 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.310302 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:28:00.311675 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.311658 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:28:00.311759 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.311705 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:00.315478 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.315420 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-110.ec2.internal.18e32a62c146c9bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-110.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-110.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-110.ec2.internal,},FirstTimestamp:2026-10-30 01:28:00.261392829 +0000 UTC m=+0.382040163,LastTimestamp:2026-10-30 01:28:00.261392829 +0000 UTC m=+0.382040163,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-110.ec2.internal,}" Oct 30 01:28:00.318241 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.318225 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnr8d" Oct 30 01:28:00.421813 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.410569 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:00.421813 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.411372 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:00.421813 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.411398 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:00.421813 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.411408 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:00.421813 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.411439 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.433280 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.433144 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.433388 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.433288 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-110.ec2.internal\": node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:00.450327 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.450295 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:28:00.451493 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.451474 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:28:00.451493 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.451494 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:28:00.451619 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.451511 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:28:00.451619 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.451565 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 01:28:00.468184 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.468169 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:00.485365 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.485347 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:00.552685 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.552663 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal"] Oct 30 01:28:00.552776 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.552718 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:00.553507 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.553491 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:00.553615 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.553521 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:00.553615 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.553556 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:00.555485 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.555470 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:00.555656 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.555635 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.555746 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.555666 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:00.556273 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.556257 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:00.556330 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.556280 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:00.556330 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.556289 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:00.556330 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.556298 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:00.556330 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.556303 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:00.556330 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.556312 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:00.558295 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.558283 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.558341 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.558305 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:00.558961 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.558949 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:00.559008 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.558971 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:00.559008 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.558985 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:00.578454 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.578439 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-110.ec2.internal\" not found" node="ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.580600 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.580586 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-110.ec2.internal\" not found" node="ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.585910 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.585897 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:00.644017 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.643993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.644104 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.644032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.644104 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.644060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec075332058530a9d0af488e14c4afa0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" (UID: \"ec075332058530a9d0af488e14c4afa0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.686928 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.686880 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:00.744826 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.744800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.744947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.744846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.744947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.744870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec075332058530a9d0af488e14c4afa0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" (UID: \"ec075332058530a9d0af488e14c4afa0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.744947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.744892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.744947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.744902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec075332058530a9d0af488e14c4afa0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" (UID: \"ec075332058530a9d0af488e14c4afa0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.744947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.744911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.787194 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.787170 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:00.881626 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.881604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.883595 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:00.883580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:00.887614 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.887598 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:00.988351 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:00.988289 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:01.088731 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:01.088706 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:01.155796 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.155778 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:01.171853 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.171828 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 01:28:01.171976 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.171957 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:28:01.172037 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.171989 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:28:01.189130 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:01.189102 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:01.289908 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:01.289881 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:01.326017 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.325985 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:23:01 +0000 UTC" deadline="2028-04-04 23:37:37.775705673 +0000 UTC" Oct 30 01:28:01.326017 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.326016 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12550h9m36.449693831s" Oct 30 01:28:01.389987 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:01.389950 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:01.399182 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.399161 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:01.400705 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.400686 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:01.400885 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.400726 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:01.400885 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.400742 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:01.491111 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:01.491085 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:01.516285 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:01.516223 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec075332058530a9d0af488e14c4afa0.slice/crio-a050cd3b4e5df5ce962f0676b2f2386ef0e8184b58a3943f81bc084779c0126d WatchSource:0}: Error finding container a050cd3b4e5df5ce962f0676b2f2386ef0e8184b58a3943f81bc084779c0126d: Status 404 returned error can't find the container with id a050cd3b4e5df5ce962f0676b2f2386ef0e8184b58a3943f81bc084779c0126d Oct 30 01:28:01.516721 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:01.516690 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573fcfb4d98e5875177cb8f4421e39f7.slice/crio-504c278b298d00620d18e633174d51c060a7cb17ff206253943c4dbd1d57a5e4 WatchSource:0}: Error finding container 504c278b298d00620d18e633174d51c060a7cb17ff206253943c4dbd1d57a5e4: Status 404 returned error can't find the container with id 504c278b298d00620d18e633174d51c060a7cb17ff206253943c4dbd1d57a5e4 Oct 30 01:28:01.521487 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.521466 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:28:01.591891 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:01.591864 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:01.660132 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.660110 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:01.690696 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.690671 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:01.741597 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.741572 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:01.770964 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.770913 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:01.771909 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.771896 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:01.795950 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:01.795935 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:02.107289 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.107216 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:02.215542 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.215496 2576 apiserver.go:52] "Watching apiserver" Oct 30 01:28:02.241569 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.241541 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 01:28:02.250023 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.250000 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:28:02.250430 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.250408 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xknfd","openshift-image-registry/node-ca-8kspl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal","openshift-network-operator/iptables-alerter-km427","openshift-ovn-kubernetes/ovnkube-node-9w4ql","kube-system/konnectivity-agent-5rdzj","kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq","openshift-dns/node-resolver-bv8wk","openshift-multus/multus-mf4tj","openshift-network-diagnostics/network-check-target-hsc2x","openshift-multus/multus-additional-cni-plugins-8bdx4","openshift-multus/network-metrics-daemon-gd5qr","openshift-cluster-node-tuning-operator/tuned-6vsqw"] Oct 30 01:28:02.253129 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.253106 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.255530 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.255509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:02.255646 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.255626 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:02.258001 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.257980 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.260347 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.260327 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.263657 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.262941 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.266294 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.265862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:02.267877 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.267860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.270910 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.270804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.271856 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.271833 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkdgr\"" Oct 30 01:28:02.272027 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.271999 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:28:02.272107 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.272098 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:28:02.272185 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.272162 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:28:02.272311 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.272294 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:28:02.272892 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.272873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.275419 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.275402 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.275546 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.275467 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:02.277959 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.277939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:02.278062 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.278011 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:02.280929 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.280481 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.302791 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.302767 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:28:02.302882 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.302837 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z9lkk" Oct 30 01:28:02.343154 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.343131 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:28:02.354413 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-system-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.354519 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-netd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.354519 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22ff8c4e-b07d-417b-b4d3-61560b014b77-hosts-file\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.354519 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jm4t\" (UniqueName: \"kubernetes.io/projected/fdd98c23-760f-487d-b8d2-be0684665b55-kube-api-access-8jm4t\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:02.354728 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-lib-modules\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.354728 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-host\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.354728 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2qx7\" (UniqueName: \"kubernetes.io/projected/2160cefb-5ed3-4ee9-abc4-9184f7825317-kube-api-access-q2qx7\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.354840 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e3639d2-bb67-4554-b567-be193af33780-host-slash\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.354840 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-hostroot\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.354840 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-conf-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.354951 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-run\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.354951 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-kubelet-config\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.354951 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-slash\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.355068 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovn-node-metrics-cert\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.355068 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.354996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7p77\" (UniqueName: \"kubernetes.io/projected/6e60155d-a79a-453f-a17a-902840cdeb14-kube-api-access-c7p77\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.355068 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355022 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-conf\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.355068 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.355218 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-var-lib-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.355218 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-config\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.355218 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-env-overrides\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.355218 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-systemd\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.355218 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf8cbe6-582e-408b-85a9-d969e18b4682-host\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.355459 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ql9j\" (UniqueName: \"kubernetes.io/projected/acf8cbe6-582e-408b-85a9-d969e18b4682-kube-api-access-4ql9j\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.355459 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-socket-dir-parent\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.355459 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-bin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.355459 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-netns\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.355624 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-socket-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.355624 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-registration-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.355624 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-tuned\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.355742 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-tmp\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.355742 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-kubelet\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.355742 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-log-socket\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.355835 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9vt\" (UniqueName: \"kubernetes.io/projected/5cafa88a-427e-45c7-a706-d56fd5425610-kube-api-access-bd9vt\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.355835 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdvd\" (UniqueName: \"kubernetes.io/projected/08cfb9ab-35cb-4695-8758-71f71f0c34c3-kube-api-access-tjdvd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.355911 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ff8c4e-b07d-417b-b4d3-61560b014b77-tmp-dir\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.355911 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355859 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-sys-fs\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.355911 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-var-lib-kubelet\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.356030 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.355913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrtd\" (UniqueName: \"kubernetes.io/projected/3e3639d2-bb67-4554-b567-be193af33780-kube-api-access-wvrtd\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.356030 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.356115 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-multus-certs\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.356204 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44ef0b6d-c29b-4925-b06f-5136ef9aa906-agent-certs\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:02.356204 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:02.356204 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-etc-kubernetes\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.356362 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-bin\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.356362 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-kubernetes\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.356362 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.356362 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-binary-copy\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.356362 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-cni-binary-copy\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.356362 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-multus-daemon-config\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.356574 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xrd\" (UniqueName: \"kubernetes.io/projected/53172623-6cce-4589-91c3-811476ab540d-kube-api-access-d9xrd\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.356574 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-systemd-units\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.356574 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.356758 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:02.356758 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysconfig\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.356758 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acf8cbe6-582e-408b-85a9-d969e18b4682-serviceca\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.356758 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-netns\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.356960 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-kubelet\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.356960 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-systemd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.356960 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.357116 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.356969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-system-cni-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.357116 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.357116 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-device-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.357116 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-modprobe-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.357116 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-dbus\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.357392 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.357392 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-cnibin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.357392 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-k8s-cni-cncf-io\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.357392 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-etc-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.357392 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-cnibin\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-ovn\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-node-log\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-script-lib\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44ef0b6d-c29b-4925-b06f-5136ef9aa906-konnectivity-ca\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-sys\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.357658 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e3639d2-bb67-4554-b567-be193af33780-iptables-alerter-script\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.358182 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-multus\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.358182 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.357709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddck4\" (UniqueName: \"kubernetes.io/projected/22ff8c4e-b07d-417b-b4d3-61560b014b77-kube-api-access-ddck4\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.381387 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381363 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:02.381387 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381363 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:02.381597 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381400 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:28:02.381597 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381445 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:28:02.381597 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381546 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:28:02.381758 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381637 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:28:02.381758 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381679 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:28:02.381893 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381874 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2gfdm\"" Oct 30 01:28:02.381955 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381916 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:28:02.381955 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.381923 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:28:02.382060 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382045 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglfc\"" Oct 30 01:28:02.382201 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382175 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:28:02.382201 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382198 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382200 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rwcx9\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382242 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382251 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382244 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lm6wv\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382385 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7p9xn\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382409 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382416 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382424 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382441 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:28:02.382444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382385 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:28:02.382947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382455 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:28:02.382947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382470 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:02.382947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382473 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7lll\"" Oct 30 01:28:02.382947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382492 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:28:02.382947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382510 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c6clm\"" Oct 30 01:28:02.382947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382513 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lzbd2\"" Oct 30 01:28:02.382947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382582 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:28:02.382947 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.382596 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:28:02.456703 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.456622 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" event={"ID":"ec075332058530a9d0af488e14c4afa0","Type":"ContainerStarted","Data":"a050cd3b4e5df5ce962f0676b2f2386ef0e8184b58a3943f81bc084779c0126d"} Oct 30 01:28:02.457810 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.457778 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" event={"ID":"573fcfb4d98e5875177cb8f4421e39f7","Type":"ContainerStarted","Data":"504c278b298d00620d18e633174d51c060a7cb17ff206253943c4dbd1d57a5e4"} Oct 30 01:28:02.457919 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.457879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-netns\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.457919 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.457903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-socket-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.458036 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.457920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-registration-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.458036 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.457937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-tuned\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.458036 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.457958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-tmp\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.458194 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458037 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-registration-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.458194 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458037 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-netns\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.458194 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458077 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-socket-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.458194 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-kubelet\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.458194 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-log-socket\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.458194 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9vt\" (UniqueName: \"kubernetes.io/projected/5cafa88a-427e-45c7-a706-d56fd5425610-kube-api-access-bd9vt\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.458194 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdvd\" (UniqueName: \"kubernetes.io/projected/08cfb9ab-35cb-4695-8758-71f71f0c34c3-kube-api-access-tjdvd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ff8c4e-b07d-417b-b4d3-61560b014b77-tmp-dir\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458213 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-kubelet\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458229 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-sys-fs\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458253 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-log-socket\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-var-lib-kubelet\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrtd\" (UniqueName: \"kubernetes.io/projected/3e3639d2-bb67-4554-b567-be193af33780-kube-api-access-wvrtd\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-multus-certs\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44ef0b6d-c29b-4925-b06f-5136ef9aa906-agent-certs\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-etc-kubernetes\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-bin\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458504 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-multus-certs\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.458533 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458533 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-bin\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.458670 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-var-lib-kubelet\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-kubernetes\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.458775 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret podName:a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:02.958724542 +0000 UTC m=+3.079371891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret") pod "global-pull-secret-syncer-xknfd" (UID: "a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458776 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ff8c4e-b07d-417b-b4d3-61560b014b77-tmp-dir\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.458798 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458830 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-kubernetes\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-binary-copy\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.458854 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:02.95883604 +0000 UTC m=+3.079483387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458899 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-sys-fs\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.458960 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-etc-kubernetes\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459052 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-cni-binary-copy\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-multus-daemon-config\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xrd\" (UniqueName: \"kubernetes.io/projected/53172623-6cce-4589-91c3-811476ab540d-kube-api-access-d9xrd\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.459277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-systemd-units\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysconfig\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acf8cbe6-582e-408b-85a9-d969e18b4682-serviceca\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-netns\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-kubelet\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-systemd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-system-cni-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-device-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459580 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-kubelet\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-device-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-modprobe-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459689 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-systemd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-dbus\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.460135 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459753 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-cnibin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459807 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459841 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-systemd-units\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459897 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-modprobe-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459921 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-system-cni-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.459957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysconfig\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-netns\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460072 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-cnibin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460209 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-binary-copy\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460506 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-cni-binary-copy\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460537 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acf8cbe6-582e-408b-85a9-d969e18b4682-serviceca\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-k8s-cni-cncf-io\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-etc-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460656 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-multus-daemon-config\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-cnibin\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.460985 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460742 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-dbus\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-etc-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460791 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-cnibin\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-k8s-cni-cncf-io\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460815 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-ovn\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-node-log\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.460970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-script-lib\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461031 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-ovn\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44ef0b6d-c29b-4925-b06f-5136ef9aa906-konnectivity-ca\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461112 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-node-log\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-sys\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.461844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461405 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e3639d2-bb67-4554-b567-be193af33780-iptables-alerter-script\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461456 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-multus\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461514 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-sys\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddck4\" (UniqueName: \"kubernetes.io/projected/22ff8c4e-b07d-417b-b4d3-61560b014b77-kube-api-access-ddck4\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-multus\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461685 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44ef0b6d-c29b-4925-b06f-5136ef9aa906-konnectivity-ca\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-system-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-netd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461831 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22ff8c4e-b07d-417b-b4d3-61560b014b77-hosts-file\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jm4t\" (UniqueName: \"kubernetes.io/projected/fdd98c23-760f-487d-b8d2-be0684665b55-kube-api-access-8jm4t\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461925 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-netd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.461971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-lib-modules\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-host\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2qx7\" (UniqueName: \"kubernetes.io/projected/2160cefb-5ed3-4ee9-abc4-9184f7825317-kube-api-access-q2qx7\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.462464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462067 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22ff8c4e-b07d-417b-b4d3-61560b014b77-hosts-file\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e3639d2-bb67-4554-b567-be193af33780-host-slash\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-hostroot\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-conf-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462181 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-host\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462184 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-system-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-run\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-kubelet-config\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-slash\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovn-node-metrics-cert\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462385 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-lib-modules\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462424 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-script-lib\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7p77\" (UniqueName: \"kubernetes.io/projected/6e60155d-a79a-453f-a17a-902840cdeb14-kube-api-access-c7p77\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462498 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-hostroot\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-conf\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462581 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-kubelet-config\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-var-lib-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.463302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-config\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-env-overrides\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-systemd\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf8cbe6-582e-408b-85a9-d969e18b4682-host\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462880 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-systemd\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462873 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-slash\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ql9j\" (UniqueName: \"kubernetes.io/projected/acf8cbe6-582e-408b-85a9-d969e18b4682-kube-api-access-4ql9j\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e3639d2-bb67-4554-b567-be193af33780-host-slash\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-socket-dir-parent\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462986 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-var-lib-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.462987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-bin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.463092 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.463147 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-conf-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.463226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-socket-dir-parent\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.463280 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-run\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.463299 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-env-overrides\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.463346 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-bin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.463379 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf8cbe6-582e-408b-85a9-d969e18b4682-host\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.464160 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.463397 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-conf\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.464989 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.463584 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-config\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.464989 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.464075 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e3639d2-bb67-4554-b567-be193af33780-iptables-alerter-script\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.468572 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.466818 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovn-node-metrics-cert\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.468572 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.467016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44ef0b6d-c29b-4925-b06f-5136ef9aa906-agent-certs\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:02.468572 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.467306 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-tmp\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.468928 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.468908 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-tuned\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.504063 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.504021 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdvd\" (UniqueName: \"kubernetes.io/projected/08cfb9ab-35cb-4695-8758-71f71f0c34c3-kube-api-access-tjdvd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.504894 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.504862 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrtd\" (UniqueName: \"kubernetes.io/projected/3e3639d2-bb67-4554-b567-be193af33780-kube-api-access-wvrtd\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.510072 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.510044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9vt\" (UniqueName: \"kubernetes.io/projected/5cafa88a-427e-45c7-a706-d56fd5425610-kube-api-access-bd9vt\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.512050 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.511815 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xrd\" (UniqueName: \"kubernetes.io/projected/53172623-6cce-4589-91c3-811476ab540d-kube-api-access-d9xrd\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.518576 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.518552 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2qx7\" (UniqueName: \"kubernetes.io/projected/2160cefb-5ed3-4ee9-abc4-9184f7825317-kube-api-access-q2qx7\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.519339 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.519318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddck4\" (UniqueName: \"kubernetes.io/projected/22ff8c4e-b07d-417b-b4d3-61560b014b77-kube-api-access-ddck4\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.519427 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.519370 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7p77\" (UniqueName: \"kubernetes.io/projected/6e60155d-a79a-453f-a17a-902840cdeb14-kube-api-access-c7p77\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.521853 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.521834 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:02.521963 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.521858 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:02.521963 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.521872 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2bwqn for pod openshift-network-diagnostics/network-check-target-hsc2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:02.521963 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.521950 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn podName:971584c9-502d-4dfa-beae-8d8d7ae88cb3 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:03.021930741 +0000 UTC m=+3.142578079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2bwqn" (UniqueName: "kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn") pod "network-check-target-hsc2x" (UID: "971584c9-502d-4dfa-beae-8d8d7ae88cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:02.523092 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.523005 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ql9j\" (UniqueName: \"kubernetes.io/projected/acf8cbe6-582e-408b-85a9-d969e18b4682-kube-api-access-4ql9j\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.524391 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.524358 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jm4t\" (UniqueName: \"kubernetes.io/projected/fdd98c23-760f-487d-b8d2-be0684665b55-kube-api-access-8jm4t\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:02.569535 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.569512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mf4tj" Oct 30 01:28:02.576983 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.576957 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:02.581116 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:02.581091 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53172623_6cce_4589_91c3_811476ab540d.slice/crio-b9ea3f0b439c9e28fa22a2943975ef5c6e1f577c3598a1a690b561831c0952af WatchSource:0}: Error finding container b9ea3f0b439c9e28fa22a2943975ef5c6e1f577c3598a1a690b561831c0952af: Status 404 returned error can't find the container with id b9ea3f0b439c9e28fa22a2943975ef5c6e1f577c3598a1a690b561831c0952af Oct 30 01:28:02.586743 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:02.586722 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2160cefb_5ed3_4ee9_abc4_9184f7825317.slice/crio-16be49c0bb60bf6eda92bf329a69f4e11ad9ddae87356df8e9d352eda5bf387a WatchSource:0}: Error finding container 16be49c0bb60bf6eda92bf329a69f4e11ad9ddae87356df8e9d352eda5bf387a: Status 404 returned error can't find the container with id 16be49c0bb60bf6eda92bf329a69f4e11ad9ddae87356df8e9d352eda5bf387a Oct 30 01:28:02.590520 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.590483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:02.601408 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:02.601385 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3639d2_bb67_4554_b567_be193af33780.slice/crio-cd28e5fba1f493a86f011eda2f7e7bd0033d46c91907e1870e063ea700a59417 WatchSource:0}: Error finding container cd28e5fba1f493a86f011eda2f7e7bd0033d46c91907e1870e063ea700a59417: Status 404 returned error can't find the container with id cd28e5fba1f493a86f011eda2f7e7bd0033d46c91907e1870e063ea700a59417 Oct 30 01:28:02.617090 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.617039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:02.618408 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.618372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:02.622076 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.622030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:02.625611 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.625382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:02.627967 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.627790 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:02.629281 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:02.629256 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ef0b6d_c29b_4925_b06f_5136ef9aa906.slice/crio-58efa8b6b6b953c92c4289f1160ba63e37553f2d48af5a0f206aed7a1971ac2e WatchSource:0}: Error finding container 58efa8b6b6b953c92c4289f1160ba63e37553f2d48af5a0f206aed7a1971ac2e: Status 404 returned error can't find the container with id 58efa8b6b6b953c92c4289f1160ba63e37553f2d48af5a0f206aed7a1971ac2e Oct 30 01:28:02.633643 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:02.633567 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ff8c4e_b07d_417b_b4d3_61560b014b77.slice/crio-9c20636b674ce676c941dae5a18613e2704ef4a4fa6e46ed8f356649665f9da3 WatchSource:0}: Error finding container 9c20636b674ce676c941dae5a18613e2704ef4a4fa6e46ed8f356649665f9da3: Status 404 returned error can't find the container with id 9c20636b674ce676c941dae5a18613e2704ef4a4fa6e46ed8f356649665f9da3 Oct 30 01:28:02.641622 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:02.641581 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e60155d_a79a_453f_a17a_902840cdeb14.slice/crio-84c6e07766b7ee19cabc5209685470de32cd0e0f92448ba7eaf3d31b6411bd28 WatchSource:0}: Error finding container 84c6e07766b7ee19cabc5209685470de32cd0e0f92448ba7eaf3d31b6411bd28: Status 404 returned error can't find the container with id 84c6e07766b7ee19cabc5209685470de32cd0e0f92448ba7eaf3d31b6411bd28 Oct 30 01:28:02.644598 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:02.644467 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf8cbe6_582e_408b_85a9_d969e18b4682.slice/crio-66a81e642a8bec8062afdc1fe7c39e670910e843991ee1c072443e19ed47e2a8 WatchSource:0}: Error finding container 66a81e642a8bec8062afdc1fe7c39e670910e843991ee1c072443e19ed47e2a8: Status 404 returned error can't find the container with id 66a81e642a8bec8062afdc1fe7c39e670910e843991ee1c072443e19ed47e2a8 Oct 30 01:28:02.648151 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.648131 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:02.658868 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:02.658836 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cafa88a_427e_45c7_a706_d56fd5425610.slice/crio-b1e24973d5ca651624d9c2eebbbacceb04eb96da53bb6658edeed8426dcba646 WatchSource:0}: Error finding container b1e24973d5ca651624d9c2eebbbacceb04eb96da53bb6658edeed8426dcba646: Status 404 returned error can't find the container with id b1e24973d5ca651624d9c2eebbbacceb04eb96da53bb6658edeed8426dcba646 Oct 30 01:28:02.968000 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.967970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:02.968118 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:02.968028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:02.968250 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.968152 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:02.968250 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.968154 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:02.968250 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.968214 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:03.968194907 +0000 UTC m=+4.088842246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:02.968409 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:02.968298 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret podName:a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:03.968276269 +0000 UTC m=+4.088923609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret") pod "global-pull-secret-syncer-xknfd" (UID: "a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:03.069390 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.068721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:03.069390 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.068906 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:03.069390 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.068931 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:03.069390 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.068946 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2bwqn for pod openshift-network-diagnostics/network-check-target-hsc2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:03.069390 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.069019 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn podName:971584c9-502d-4dfa-beae-8d8d7ae88cb3 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:04.068988663 +0000 UTC m=+4.189636010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bwqn" (UniqueName: "kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn") pod "network-check-target-hsc2x" (UID: "971584c9-502d-4dfa-beae-8d8d7ae88cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:03.312565 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.312429 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:23:03 +0000 UTC" deadline="2028-07-11 00:16:48.435513992 +0000 UTC" Oct 30 01:28:03.312565 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.312514 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14878h48m45.12300438s" Oct 30 01:28:03.451862 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.451822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:03.451973 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.451948 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:03.452270 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.452253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:03.452348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.452334 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:03.465209 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.465179 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" event={"ID":"5cafa88a-427e-45c7-a706-d56fd5425610","Type":"ContainerStarted","Data":"b1e24973d5ca651624d9c2eebbbacceb04eb96da53bb6658edeed8426dcba646"} Oct 30 01:28:03.475430 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.475394 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8kspl" event={"ID":"acf8cbe6-582e-408b-85a9-d969e18b4682","Type":"ContainerStarted","Data":"66a81e642a8bec8062afdc1fe7c39e670910e843991ee1c072443e19ed47e2a8"} Oct 30 01:28:03.484768 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.484732 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" event={"ID":"08cfb9ab-35cb-4695-8758-71f71f0c34c3","Type":"ContainerStarted","Data":"e628113131fd5a44909ada19ea8ae173a532d988aacb998698aff45a9fc8e424"} Oct 30 01:28:03.496881 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.496850 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5rdzj" event={"ID":"44ef0b6d-c29b-4925-b06f-5136ef9aa906","Type":"ContainerStarted","Data":"58efa8b6b6b953c92c4289f1160ba63e37553f2d48af5a0f206aed7a1971ac2e"} Oct 30 01:28:03.498674 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.498646 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-km427" event={"ID":"3e3639d2-bb67-4554-b567-be193af33780","Type":"ContainerStarted","Data":"cd28e5fba1f493a86f011eda2f7e7bd0033d46c91907e1870e063ea700a59417"} Oct 30 01:28:03.500897 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.500868 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" event={"ID":"2160cefb-5ed3-4ee9-abc4-9184f7825317","Type":"ContainerStarted","Data":"16be49c0bb60bf6eda92bf329a69f4e11ad9ddae87356df8e9d352eda5bf387a"} Oct 30 01:28:03.503299 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.503274 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" event={"ID":"6e60155d-a79a-453f-a17a-902840cdeb14","Type":"ContainerStarted","Data":"84c6e07766b7ee19cabc5209685470de32cd0e0f92448ba7eaf3d31b6411bd28"} Oct 30 01:28:03.508274 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.508247 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bv8wk" event={"ID":"22ff8c4e-b07d-417b-b4d3-61560b014b77","Type":"ContainerStarted","Data":"9c20636b674ce676c941dae5a18613e2704ef4a4fa6e46ed8f356649665f9da3"} Oct 30 01:28:03.526487 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.526463 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mf4tj" event={"ID":"53172623-6cce-4589-91c3-811476ab540d","Type":"ContainerStarted","Data":"b9ea3f0b439c9e28fa22a2943975ef5c6e1f577c3598a1a690b561831c0952af"} Oct 30 01:28:03.976236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.975902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:03.976236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:03.976172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:03.976452 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.976326 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:03.976452 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.976391 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:05.97637101 +0000 UTC m=+6.097018336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:03.976853 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.976829 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:03.976931 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:03.976893 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret podName:a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:05.976875627 +0000 UTC m=+6.097522960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret") pod "global-pull-secret-syncer-xknfd" (UID: "a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:04.037349 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:04.037317 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:04.076935 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:04.076902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:04.077126 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:04.077064 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:04.077126 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:04.077084 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:04.077126 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:04.077106 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2bwqn for pod openshift-network-diagnostics/network-check-target-hsc2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:04.077289 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:04.077159 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn podName:971584c9-502d-4dfa-beae-8d8d7ae88cb3 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:06.077142085 +0000 UTC m=+6.197789411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bwqn" (UniqueName: "kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn") pod "network-check-target-hsc2x" (UID: "971584c9-502d-4dfa-beae-8d8d7ae88cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:04.312841 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:04.312732 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:23:03 +0000 UTC" deadline="2028-05-03 15:59:25.196674068 +0000 UTC" Oct 30 01:28:04.312841 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:04.312788 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13238h31m20.883888814s" Oct 30 01:28:04.452435 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:04.451916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:04.452435 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:04.452064 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:04.556066 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:04.556026 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" event={"ID":"ec075332058530a9d0af488e14c4afa0","Type":"ContainerStarted","Data":"04ccceee214461e3118e6ca32b181d12bd6ca521f219c89bd58b368b3a55c15e"} Oct 30 01:28:04.567671 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:04.567520 2576 generic.go:356] "Generic (PLEG): container finished" podID="573fcfb4d98e5875177cb8f4421e39f7" containerID="d6f09d761f1665b63c0640d2327b81d722520a8b0e0b1ce50369bbcb651c71dc" exitCode=0 Oct 30 01:28:04.567671 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:04.567561 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" event={"ID":"573fcfb4d98e5875177cb8f4421e39f7","Type":"ContainerDied","Data":"d6f09d761f1665b63c0640d2327b81d722520a8b0e0b1ce50369bbcb651c71dc"} Oct 30 01:28:04.652697 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:04.652638 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" podStartSLOduration=2.652618455 podStartE2EDuration="2.652618455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:04.604444958 +0000 UTC m=+4.725092306" watchObservedRunningTime="2026-10-30 01:28:04.652618455 +0000 UTC m=+4.773265804" Oct 30 01:28:05.451907 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:05.451876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:05.452421 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:05.452024 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:05.452421 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:05.452121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:05.452421 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:05.452214 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:05.593971 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:05.586219 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" event={"ID":"573fcfb4d98e5875177cb8f4421e39f7","Type":"ContainerStarted","Data":"e9d20098885e10db0863bb0e4e90211d5fa88aac5bc1d17ed25c8145abc0e53d"} Oct 30 01:28:06.001870 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:06.000958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:06.001870 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:06.001004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:06.001870 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:06.001149 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:06.001870 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:06.001211 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:10.001191692 +0000 UTC m=+10.121839017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:06.001870 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:06.001283 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:06.001870 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:06.001307 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret podName:a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:10.001300008 +0000 UTC m=+10.121947331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret") pod "global-pull-secret-syncer-xknfd" (UID: "a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:06.102158 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:06.102124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:06.102370 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:06.102352 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:06.102450 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:06.102378 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:06.102450 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:06.102394 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2bwqn for pod openshift-network-diagnostics/network-check-target-hsc2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:06.102557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:06.102455 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn podName:971584c9-502d-4dfa-beae-8d8d7ae88cb3 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:10.102435445 +0000 UTC m=+10.223082772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bwqn" (UniqueName: "kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn") pod "network-check-target-hsc2x" (UID: "971584c9-502d-4dfa-beae-8d8d7ae88cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:06.452076 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:06.452045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:06.452461 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:06.452181 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:06.595161 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:06.593826 2576 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="7de466b828952e7010247fc5eb3d8ac1bc9510e85bbf8972bdf51551693c1ce5" exitCode=0 Oct 30 01:28:06.595161 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:06.594952 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" event={"ID":"5cafa88a-427e-45c7-a706-d56fd5425610","Type":"ContainerDied","Data":"7de466b828952e7010247fc5eb3d8ac1bc9510e85bbf8972bdf51551693c1ce5"} Oct 30 01:28:06.691713 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:06.690393 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" podStartSLOduration=4.690373606 podStartE2EDuration="4.690373606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:05.636805288 +0000 UTC m=+5.757452634" watchObservedRunningTime="2026-10-30 01:28:06.690373606 +0000 UTC m=+6.811020953" Oct 30 01:28:07.453346 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:07.452603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:07.453346 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:07.452744 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:07.453346 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:07.453178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:07.453346 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:07.453294 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:07.600456 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:07.600415 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5rdzj" event={"ID":"44ef0b6d-c29b-4925-b06f-5136ef9aa906","Type":"ContainerStarted","Data":"0c6773f7a04581340367f4cfb858d0b5ee143e513f8532a44abcc497b62ef299"} Oct 30 01:28:08.454913 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:08.453738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:08.454913 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:08.453870 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:08.606202 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:08.606171 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8kspl" event={"ID":"acf8cbe6-582e-408b-85a9-d969e18b4682","Type":"ContainerStarted","Data":"3ae911d9ffd707e7789ce426c94d277a43f7099caba15f682164231b8c460990"} Oct 30 01:28:08.670306 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:08.669514 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5rdzj" podStartSLOduration=3.871941209 podStartE2EDuration="7.669495289s" totalImagesPullingTime="3.79755408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:07.649154099 +0000 UTC m=+7.769801449" watchObservedRunningTime="2026-10-30 01:28:08.669495289 +0000 UTC m=+8.790142638" Oct 30 01:28:09.452058 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:09.452019 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:09.452267 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:09.452172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:09.452575 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:09.452538 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:09.452690 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:09.452674 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:09.612779 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:09.612742 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" event={"ID":"6e60155d-a79a-453f-a17a-902840cdeb14","Type":"ContainerStarted","Data":"676bc128f342293ec9d7162d12d3bd5ffb0658227e6b2efaeb34641db6089336"} Oct 30 01:28:10.040360 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:10.040319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:10.040593 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:10.040377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:10.040593 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:10.040553 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:10.040715 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:10.040637 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:18.040617776 +0000 UTC m=+18.161265125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:10.041087 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:10.041063 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:10.041220 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:10.041147 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret podName:a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:18.041112474 +0000 UTC m=+18.161759822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret") pod "global-pull-secret-syncer-xknfd" (UID: "a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:10.141973 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:10.141255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:10.141973 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:10.141451 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:10.141973 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:10.141470 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:10.141973 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:10.141484 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2bwqn for pod openshift-network-diagnostics/network-check-target-hsc2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:10.141973 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:10.141553 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn podName:971584c9-502d-4dfa-beae-8d8d7ae88cb3 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:18.141529199 +0000 UTC m=+18.262176547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bwqn" (UniqueName: "kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn") pod "network-check-target-hsc2x" (UID: "971584c9-502d-4dfa-beae-8d8d7ae88cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:10.453540 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:10.453474 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:10.453702 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:10.453666 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:10.515889 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:10.515859 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:28:10.619620 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:10.619439 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" event={"ID":"6e60155d-a79a-453f-a17a-902840cdeb14","Type":"ContainerStarted","Data":"5bd51707ff7ea6ae5c2229b54e98e42e9589332504acba286e4178424ac9a9ea"} Oct 30 01:28:10.677237 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:10.677112 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:10.677950 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:10.677673 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:10.725790 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:10.725288 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8kspl" podStartSLOduration=4.308094576 podStartE2EDuration="9.725272238s" totalImagesPullingTime="5.417177662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:08.670293726 +0000 UTC m=+8.790941049" watchObservedRunningTime="2026-10-30 01:28:10.725272238 +0000 UTC m=+10.845919587" Oct 30 01:28:11.316909 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.316662 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:28:10.515887017Z","UUID":"0b4d4e8c-7ce0-4163-931c-6b8b70830c53","Handler":null,"Name":"","Endpoint":""} Oct 30 01:28:11.318220 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.318196 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:28:11.318339 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.318229 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:28:11.452397 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.452322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:11.452397 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.452364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:11.452585 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:11.452473 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:11.452657 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:11.452615 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:11.626705 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.626671 2576 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="f9b8f0dc558ae76ea19b9b4a64efdb8840f780199e948704af73fb664faf6f41" exitCode=0 Oct 30 01:28:11.627102 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.626774 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" event={"ID":"5cafa88a-427e-45c7-a706-d56fd5425610","Type":"ContainerDied","Data":"f9b8f0dc558ae76ea19b9b4a64efdb8840f780199e948704af73fb664faf6f41"} Oct 30 01:28:11.631703 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.631246 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bv8wk" event={"ID":"22ff8c4e-b07d-417b-b4d3-61560b014b77","Type":"ContainerStarted","Data":"55944fea29ded85b96c7219f6c8cd997adc7bc935b697bf6bc282b68ee3cfcfa"} Oct 30 01:28:11.633138 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.633103 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mf4tj" event={"ID":"53172623-6cce-4589-91c3-811476ab540d","Type":"ContainerStarted","Data":"c85452e32c52055c939589d778fda0d624f4f353078f932f6cfd8f5fbf3646d8"} Oct 30 01:28:11.733290 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.733197 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mf4tj" podStartSLOduration=2.640925927 podStartE2EDuration="10.733181552s" totalImagesPullingTime="8.092255625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:11.732962593 +0000 UTC m=+11.853609952" watchObservedRunningTime="2026-10-30 01:28:11.733181552 +0000 UTC m=+11.853828895" Oct 30 01:28:11.780195 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:11.780139 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bv8wk" podStartSLOduration=2.364362843 podStartE2EDuration="10.780122172s" totalImagesPullingTime="8.415759329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:11.779553587 +0000 UTC m=+11.900200934" watchObservedRunningTime="2026-10-30 01:28:11.780122172 +0000 UTC m=+11.900769519" Oct 30 01:28:12.452378 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:12.452341 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:12.452631 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:12.452469 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:12.637435 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:12.637396 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-km427" event={"ID":"3e3639d2-bb67-4554-b567-be193af33780","Type":"ContainerStarted","Data":"20b48bc181178e0a60b42370ca940e9d68e1f3b412fa66ed3e633ad044e8f631"} Oct 30 01:28:12.640733 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:12.640704 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" event={"ID":"6e60155d-a79a-453f-a17a-902840cdeb14","Type":"ContainerStarted","Data":"1d6d72d30cfb60a14e927348b907a7f408ced5ae834e14531d3c9b5fc1e5d234"} Oct 30 01:28:12.680799 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:12.680752 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-km427" podStartSLOduration=3.357859171 podStartE2EDuration="11.680740686s" totalImagesPullingTime="8.322881515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:12.680603837 +0000 UTC m=+12.801251180" watchObservedRunningTime="2026-10-30 01:28:12.680740686 +0000 UTC m=+12.801388028" Oct 30 01:28:12.727445 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:12.727354 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" podStartSLOduration=3.2538817079999998 podStartE2EDuration="11.727337277s" totalImagesPullingTime="8.473455569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:12.726775586 +0000 UTC m=+12.847422930" watchObservedRunningTime="2026-10-30 01:28:12.727337277 +0000 UTC m=+12.847984623" Oct 30 01:28:13.451935 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:13.451900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:13.452085 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:13.451913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:13.452085 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:13.452036 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:13.452205 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:13.452113 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:13.644627 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:13.644545 2576 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="d7d6d21ae66d78357bcbb0cababff4f090e634e845b319c09f3dea53c52f8948" exitCode=0 Oct 30 01:28:13.645301 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:13.644633 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" event={"ID":"5cafa88a-427e-45c7-a706-d56fd5425610","Type":"ContainerDied","Data":"d7d6d21ae66d78357bcbb0cababff4f090e634e845b319c09f3dea53c52f8948"} Oct 30 01:28:13.647633 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:13.647584 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" event={"ID":"2160cefb-5ed3-4ee9-abc4-9184f7825317","Type":"ContainerStarted","Data":"9915d8471ba3a31eb4ff63b1a7c9fd0663903fc49fd0ab22488d843697343aba"} Oct 30 01:28:13.741001 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:13.740942 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" podStartSLOduration=2.012507211 podStartE2EDuration="12.740924832s" totalImagesPullingTime="10.728417621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:13.740029846 +0000 UTC m=+13.860677212" watchObservedRunningTime="2026-10-30 01:28:13.740924832 +0000 UTC m=+13.861572176" Oct 30 01:28:14.451997 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:14.451964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:14.452175 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:14.452076 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:14.652829 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:14.652424 2576 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="e1df9d224c260599eb8bce42bdc3a31e6f46b69b6689eb0fd51adce29ded89ec" exitCode=0 Oct 30 01:28:14.652829 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:14.652503 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" event={"ID":"5cafa88a-427e-45c7-a706-d56fd5425610","Type":"ContainerDied","Data":"e1df9d224c260599eb8bce42bdc3a31e6f46b69b6689eb0fd51adce29ded89ec"} Oct 30 01:28:15.451936 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:15.451895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:15.451936 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:15.451941 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:15.452147 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:15.452030 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:15.452322 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:15.452290 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:16.451984 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:16.451949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:16.452432 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:16.452110 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:16.659737 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:16.659700 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" event={"ID":"5cafa88a-427e-45c7-a706-d56fd5425610","Type":"ContainerStarted","Data":"3363270155a2a608647c6e7305e74c424b6fc7d92682e0db0709c6ec88697aae"} Oct 30 01:28:17.452266 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:17.452238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:17.452671 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:17.452357 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:17.452671 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:17.452401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:17.452671 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:17.452474 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:17.664136 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:17.664103 2576 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="3363270155a2a608647c6e7305e74c424b6fc7d92682e0db0709c6ec88697aae" exitCode=0 Oct 30 01:28:17.664288 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:17.664149 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" event={"ID":"5cafa88a-427e-45c7-a706-d56fd5425610","Type":"ContainerDied","Data":"3363270155a2a608647c6e7305e74c424b6fc7d92682e0db0709c6ec88697aae"} Oct 30 01:28:18.115994 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.115973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:18.116091 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.116008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:18.116155 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:18.116112 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:18.116155 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:18.116115 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:18.116238 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:18.116170 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.116152533 +0000 UTC m=+34.236799868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:18.116238 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:18.116189 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret podName:a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.116179413 +0000 UTC m=+34.236826734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret") pod "global-pull-secret-syncer-xknfd" (UID: "a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:18.216625 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.216569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:18.216709 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:18.216681 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:18.216709 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:18.216694 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:18.216709 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:18.216703 2576 projected.go:196] Error preparing data for projected volume kube-api-access-2bwqn for pod openshift-network-diagnostics/network-check-target-hsc2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:18.216801 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:18.216744 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn podName:971584c9-502d-4dfa-beae-8d8d7ae88cb3 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.216732523 +0000 UTC m=+34.337379857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bwqn" (UniqueName: "kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn") pod "network-check-target-hsc2x" (UID: "971584c9-502d-4dfa-beae-8d8d7ae88cb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:18.452307 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.452279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:18.452705 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:18.452370 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:18.668814 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.668735 2576 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="92d28c5aa8b08e4e5bef9518cb06842fa6c8e58105de92459f57a0e1a6707dce" exitCode=0 Oct 30 01:28:18.668932 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.668816 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" event={"ID":"5cafa88a-427e-45c7-a706-d56fd5425610","Type":"ContainerDied","Data":"92d28c5aa8b08e4e5bef9518cb06842fa6c8e58105de92459f57a0e1a6707dce"} Oct 30 01:28:18.674154 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.674121 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" event={"ID":"08cfb9ab-35cb-4695-8758-71f71f0c34c3","Type":"ContainerStarted","Data":"2e6026bebfef5137294d643239d81ebacd7f02d572b4d664b886f928f4d01c97"} Oct 30 01:28:18.674272 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.674254 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" event={"ID":"08cfb9ab-35cb-4695-8758-71f71f0c34c3","Type":"ContainerStarted","Data":"b86f4247466eb98b72127fcdae949069879e020fd8d48d85c7e84fc9b4796cfb"} Oct 30 01:28:18.674342 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.674320 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" event={"ID":"08cfb9ab-35cb-4695-8758-71f71f0c34c3","Type":"ContainerStarted","Data":"35873391e4b312b23e6ff501be974f04f89a67d4bda0eea48a7cbde807554b42"} Oct 30 01:28:18.674390 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.674381 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" event={"ID":"08cfb9ab-35cb-4695-8758-71f71f0c34c3","Type":"ContainerStarted","Data":"c40e637a36441ff808977c55e2c73b74918a9ced9351cf44f29c5ac5da7e486c"} Oct 30 01:28:18.674454 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.674439 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" event={"ID":"08cfb9ab-35cb-4695-8758-71f71f0c34c3","Type":"ContainerStarted","Data":"53ac3c09554c8cfc0fb2b4cef7c43671b65c50be391029e86f887ea064c7989f"} Oct 30 01:28:18.674666 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:18.674587 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" event={"ID":"08cfb9ab-35cb-4695-8758-71f71f0c34c3","Type":"ContainerStarted","Data":"e48d3f18f739883fd54529bc287f7313805d0cc977d81afd2a40afd2afe2b15d"} Oct 30 01:28:19.452554 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:19.452499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:19.452554 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:19.452553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:19.453124 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:19.452680 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:19.453124 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:19.452784 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:19.679027 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:19.678996 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" event={"ID":"5cafa88a-427e-45c7-a706-d56fd5425610","Type":"ContainerStarted","Data":"96bcdf23877af7472957dd6200cc52f4f63ea6a0762f910728a93e9304712bfd"} Oct 30 01:28:19.773768 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:19.773681 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8bdx4" podStartSLOduration=7.174329912 podStartE2EDuration="18.773668166s" totalImagesPullingTime="11.599338254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:19.759666988 +0000 UTC m=+19.880314328" watchObservedRunningTime="2026-10-30 01:28:19.773668166 +0000 UTC m=+19.894315563" Oct 30 01:28:20.453197 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:20.453165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:20.453656 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:20.453250 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:20.683471 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:20.683392 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" event={"ID":"08cfb9ab-35cb-4695-8758-71f71f0c34c3","Type":"ContainerStarted","Data":"fc60cdd990d981271bc6eb662fa5ff09ad41d5d5a343794831f07891c71cae30"} Oct 30 01:28:20.720047 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:20.720027 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:20.720129 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:20.720119 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:20.720710 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:20.720696 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:21.451857 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:21.451827 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:21.451857 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:21.451864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:21.452078 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:21.451934 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:21.452078 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:21.452001 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:22.452116 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:22.452089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:22.452747 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:22.452193 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:22.690232 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:22.690153 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" event={"ID":"08cfb9ab-35cb-4695-8758-71f71f0c34c3","Type":"ContainerStarted","Data":"39ef49709eec830388b0e2ed17929664e4200c6660124a88132840fd11ee431b"} Oct 30 01:28:22.690472 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:22.690448 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:22.690635 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:22.690509 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:22.705458 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:22.705439 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:22.790043 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:22.790002 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" podStartSLOduration=6.518154218 podStartE2EDuration="21.789989567s" totalImagesPullingTime="15.271835349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:22.78965296 +0000 UTC m=+22.910300303" watchObservedRunningTime="2026-10-30 01:28:22.789989567 +0000 UTC m=+22.910636910" Oct 30 01:28:23.452674 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:23.452499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:23.453089 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:23.452546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:23.453089 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:23.452746 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:23.453089 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:23.452832 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:23.693303 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:23.693265 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:23.709067 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:23.709008 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:24.383027 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:24.382998 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xknfd"] Oct 30 01:28:24.383276 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:24.383106 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:24.383276 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:24.383211 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:24.388026 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:24.388000 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gd5qr"] Oct 30 01:28:24.388157 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:24.388092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:24.388217 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:24.388172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:24.388585 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:24.388567 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hsc2x"] Oct 30 01:28:24.388681 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:24.388668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:24.388776 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:24.388759 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:26.452254 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:26.452232 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:26.452633 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:26.452255 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:26.452633 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:26.452326 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:26.452633 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:26.452434 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:26.452633 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:26.452489 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:26.452633 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:26.452593 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:28.452706 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:28.452676 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:28.452706 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:28.452696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:28.453245 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:28.452679 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:28.453245 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:28.452800 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xknfd" podUID="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" Oct 30 01:28:28.453245 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:28.452855 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gd5qr" podUID="fdd98c23-760f-487d-b8d2-be0684665b55" Oct 30 01:28:28.453245 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:28.452924 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hsc2x" podUID="971584c9-502d-4dfa-beae-8d8d7ae88cb3" Oct 30 01:28:30.195176 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.195152 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeReady" Oct 30 01:28:30.195505 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.195253 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 01:28:30.252641 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.252612 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z42rk"] Oct 30 01:28:30.255822 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.255802 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jlj8q"] Oct 30 01:28:30.255969 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.255951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z42rk" Oct 30 01:28:30.259152 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.259131 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6sdlf"] Oct 30 01:28:30.259284 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.259260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" Oct 30 01:28:30.262211 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.262183 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t"] Oct 30 01:28:30.262312 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.262283 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:30.268285 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.268266 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx"] Oct 30 01:28:30.268398 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.268381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" Oct 30 01:28:30.270929 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.270910 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rzw29"] Oct 30 01:28:30.271046 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.271029 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:30.274155 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.274138 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9659j"] Oct 30 01:28:30.274278 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.274261 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.276880 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.276864 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75f7cfffd5-4l7bn"] Oct 30 01:28:30.276975 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.276961 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.279653 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.279638 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vg24q"] Oct 30 01:28:30.279759 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.279738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.282813 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.282794 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h"] Oct 30 01:28:30.283285 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.283266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vg24q" Oct 30 01:28:30.286576 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.286558 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj"] Oct 30 01:28:30.287104 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.287082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.289569 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.289551 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6976fcd64d-82clx"] Oct 30 01:28:30.289646 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.289596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.290944 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.290918 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.291116 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.291101 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.292265 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.292242 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc"] Oct 30 01:28:30.292361 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.292307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.294253 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.294233 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jnw2v\"" Oct 30 01:28:30.294371 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.294303 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-sn6bf\"" Oct 30 01:28:30.295464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.295444 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f49dfcbf7-ssq6d"] Oct 30 01:28:30.295591 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.295576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" Oct 30 01:28:30.296008 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.295988 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 01:28:30.299793 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.299775 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7"] Oct 30 01:28:30.299943 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.299928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.302787 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.302770 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dvvl4"] Oct 30 01:28:30.302904 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.302888 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:30.308231 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.306603 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t"] Oct 30 01:28:30.308231 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.306643 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z42rk"] Oct 30 01:28:30.308231 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.306668 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zmb2x"] Oct 30 01:28:30.308231 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.306949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.316796 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.316774 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6sdlf"] Oct 30 01:28:30.316895 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.316822 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vg24q"] Oct 30 01:28:30.316895 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.316846 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx"] Oct 30 01:28:30.316895 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.316876 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rzw29"] Oct 30 01:28:30.316895 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.316897 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9659j"] Oct 30 01:28:30.317105 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.316946 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75f7cfffd5-4l7bn"] Oct 30 01:28:30.317105 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.316970 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jlj8q"] Oct 30 01:28:30.317105 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.317017 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6976fcd64d-82clx"] Oct 30 01:28:30.317105 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.317074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:30.317398 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.317377 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7"] Oct 30 01:28:30.322778 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.322558 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj"] Oct 30 01:28:30.324206 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.324188 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f49dfcbf7-ssq6d"] Oct 30 01:28:30.325147 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.325128 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h"] Oct 30 01:28:30.334444 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.334399 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc"] Oct 30 01:28:30.342417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.342397 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dvvl4"] Oct 30 01:28:30.347787 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.347765 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zmb2x"] Oct 30 01:28:30.387110 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.387093 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.387572 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.387560 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.387648 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.387579 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 01:28:30.387705 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.387653 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 01:28:30.388016 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.387998 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.388205 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388191 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.388277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388244 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.388277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388247 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 01:28:30.388355 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388295 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 01:28:30.388408 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388358 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.388408 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388388 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.388408 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388397 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.388520 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388505 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.388593 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388580 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.388648 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388597 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.388648 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388614 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 01:28:30.388737 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388699 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.388737 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388708 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 01:28:30.388737 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388710 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.388737 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.388716 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.405291 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad411a5-bb79-4e0c-abd3-85c5ba535c93-tmp-dir\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.405396 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.405396 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aad411a5-bb79-4e0c-abd3-85c5ba535c93-config-volume\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.405396 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.405396 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:30.405560 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.405560 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zsmr\" (UniqueName: \"kubernetes.io/projected/3c903b8f-507e-4dec-801e-3213298d7e0a-kube-api-access-7zsmr\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.405560 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:30.405560 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xfg\" (UniqueName: \"kubernetes.io/projected/b82d0410-941f-43e4-9ae9-5d37f9c409be-kube-api-access-79xfg\") pod \"network-check-source-6f78d59f99-jqlwc\" (UID: \"b82d0410-941f-43e4-9ae9-5d37f9c409be\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" Oct 30 01:28:30.405560 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.405560 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx95x\" (UniqueName: \"kubernetes.io/projected/aad411a5-bb79-4e0c-abd3-85c5ba535c93-kube-api-access-bx95x\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.405760 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a968163-1344-4372-a8b9-bf896d90e6a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.405760 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqx8q\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.405760 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.405760 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-config\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.405760 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9xkz\" (UniqueName: \"kubernetes.io/projected/6a968163-1344-4372-a8b9-bf896d90e6a9-kube-api-access-x9xkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.405760 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c903b8f-507e-4dec-801e-3213298d7e0a-serving-cert\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.405961 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-stats-auth\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.405961 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gsmz\" (UniqueName: \"kubernetes.io/projected/34fec9a2-a544-4181-bf3a-b3e81ea7929f-kube-api-access-6gsmz\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:30.405961 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.405961 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.405961 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-default-certificate\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.405961 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.405958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp748\" (UniqueName: \"kubernetes.io/projected/a8ac7159-dcc1-46dc-bae0-eae434811dbe-kube-api-access-gp748\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.406196 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc3b9845-9528-42c3-bdae-dc73fec82643-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:30.406196 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.406196 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.406196 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2hb2\" (UniqueName: \"kubernetes.io/projected/eadbf8a4-af76-4fab-b804-13e1ac18afcc-kube-api-access-f2hb2\") pod \"kas-connection-checker-685895c7c9-z42rk\" (UID: \"eadbf8a4-af76-4fab-b804-13e1ac18afcc\") " pod="kube-system/kas-connection-checker-685895c7c9-z42rk" Oct 30 01:28:30.406196 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8ac7159-dcc1-46dc-bae0-eae434811dbe-config\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.406196 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8m9r\" (UniqueName: \"kubernetes.io/projected/bd24e16c-c31b-4af0-b518-cfe8c85937ac-kube-api-access-t8m9r\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:30.406417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-tmp\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.406417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrh86\" (UniqueName: \"kubernetes.io/projected/7c481c3b-feae-4510-83c5-0dc08ce51db0-kube-api-access-mrh86\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.406417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.406417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-service-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.406417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtlvp\" (UniqueName: \"kubernetes.io/projected/ca1fbc85-1496-4e79-9bc9-330a8540a39e-kube-api-access-rtlvp\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:30.406417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-trusted-ca\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.406417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.406417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fgwl\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.406712 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qsv9\" (UniqueName: \"kubernetes.io/projected/2986b483-9c1e-4d59-89d5-01e287807f8a-kube-api-access-6qsv9\") pod \"volume-data-source-validator-58c9f5f46d-srr2t\" (UID: \"2986b483-9c1e-4d59-89d5-01e287807f8a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" Oct 30 01:28:30.406712 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:30.406712 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c481c3b-feae-4510-83c5-0dc08ce51db0-serving-cert\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.406712 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a968163-1344-4372-a8b9-bf896d90e6a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.406712 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.406712 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.406712 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.406712 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34fec9a2-a544-4181-bf3a-b3e81ea7929f-cert\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:30.406712 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72c7\" (UniqueName: \"kubernetes.io/projected/02ab5bdf-56b9-4176-af0e-f41d92760b84-kube-api-access-f72c7\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca1fbc85-1496-4e79-9bc9-330a8540a39e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-snapshots\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntm6\" (UniqueName: \"kubernetes.io/projected/53abc737-fbef-4574-96b8-b9adee4fade2-kube-api-access-7ntm6\") pod \"kas-connection-checker-685895c7c9-vg24q\" (UID: \"53abc737-fbef-4574-96b8-b9adee4fade2\") " pod="kube-system/kas-connection-checker-685895c7c9-vg24q" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8ac7159-dcc1-46dc-bae0-eae434811dbe-serving-cert\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.406958 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.406953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78x8b\" (UniqueName: \"kubernetes.io/projected/c36ce70e-acde-4698-89b9-37d85639a669-kube-api-access-78x8b\") pod \"kas-connection-checker-685895c7c9-jlj8q\" (UID: \"c36ce70e-acde-4698-89b9-37d85639a669\") " pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" Oct 30 01:28:30.417561 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.417167 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.417561 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.417217 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.417561 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.417416 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:28:30.417561 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.417493 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.417957 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.418600 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.419190 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f29g6\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.419261 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gx9tc\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.419193 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qfrfj\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.419446 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.419492 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.419813 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.419980 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420024 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420028 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sms4c\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420070 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420195 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cz8nh\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420275 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420515 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420548 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420603 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qxp87\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420826 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jr8lv\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420845 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2p848\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.420889 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.421134 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.421181 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.421197 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.421245 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.421370 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.421503 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 01:28:30.421550 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.421510 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 01:28:30.422598 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.422266 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-f7wwf\"" Oct 30 01:28:30.422598 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.422285 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:28:30.422598 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.422295 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zqs4x\"" Oct 30 01:28:30.422598 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.422354 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:28:30.422598 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.422396 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hdprk\"" Oct 30 01:28:30.422598 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.422423 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:30.452332 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.452315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:30.452383 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.452347 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:30.452543 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.452513 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:30.491988 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.491970 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:28:30.492324 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.492310 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nc846\"" Oct 30 01:28:30.492376 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.492330 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfxzn\"" Oct 30 01:28:30.492376 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.492370 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hp2kr\"" Oct 30 01:28:30.492445 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.492378 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:28:30.507412 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-stats-auth\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.507487 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gsmz\" (UniqueName: \"kubernetes.io/projected/34fec9a2-a544-4181-bf3a-b3e81ea7929f-kube-api-access-6gsmz\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:30.507487 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.507487 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.507487 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-default-certificate\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.507487 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp748\" (UniqueName: \"kubernetes.io/projected/a8ac7159-dcc1-46dc-bae0-eae434811dbe-kube-api-access-gp748\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.507739 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.507579 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:30.507739 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.507595 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6976fcd64d-82clx: secret "image-registry-tls" not found Oct 30 01:28:30.507739 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc3b9845-9528-42c3-bdae-dc73fec82643-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:30.507739 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.507648 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls podName:8012711c-3e84-43ab-9eac-8e2280709e74 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:31.007629917 +0000 UTC m=+31.128277252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls") pod "image-registry-6976fcd64d-82clx" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74") : secret "image-registry-tls" not found Oct 30 01:28:30.507739 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.507739 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.508039 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hb2\" (UniqueName: \"kubernetes.io/projected/eadbf8a4-af76-4fab-b804-13e1ac18afcc-kube-api-access-f2hb2\") pod \"kas-connection-checker-685895c7c9-z42rk\" (UID: \"eadbf8a4-af76-4fab-b804-13e1ac18afcc\") " pod="kube-system/kas-connection-checker-685895c7c9-z42rk" Oct 30 01:28:30.508039 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.507974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8ac7159-dcc1-46dc-bae0-eae434811dbe-config\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.508039 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8m9r\" (UniqueName: \"kubernetes.io/projected/bd24e16c-c31b-4af0-b518-cfe8c85937ac-kube-api-access-t8m9r\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:30.508039 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-tmp\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.508236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrh86\" (UniqueName: \"kubernetes.io/projected/7c481c3b-feae-4510-83c5-0dc08ce51db0-kube-api-access-mrh86\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.508236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.508236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508094 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.508236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-service-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.508236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtlvp\" (UniqueName: \"kubernetes.io/projected/ca1fbc85-1496-4e79-9bc9-330a8540a39e-kube-api-access-rtlvp\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:30.508236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-trusted-ca\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.508236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.508236 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fgwl\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qsv9\" (UniqueName: \"kubernetes.io/projected/2986b483-9c1e-4d59-89d5-01e287807f8a-kube-api-access-6qsv9\") pod \"volume-data-source-validator-58c9f5f46d-srr2t\" (UID: \"2986b483-9c1e-4d59-89d5-01e287807f8a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c481c3b-feae-4510-83c5-0dc08ce51db0-serving-cert\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a968163-1344-4372-a8b9-bf896d90e6a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc3b9845-9528-42c3-bdae-dc73fec82643-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34fec9a2-a544-4181-bf3a-b3e81ea7929f-cert\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72c7\" (UniqueName: \"kubernetes.io/projected/02ab5bdf-56b9-4176-af0e-f41d92760b84-kube-api-access-f72c7\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.508630 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca1fbc85-1496-4e79-9bc9-330a8540a39e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:30.509131 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.508944 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:28:30.509131 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.508962 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-service-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.509131 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.509008 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:31.008990653 +0000 UTC m=+31.129637992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : secret "router-metrics-certs-default" not found Oct 30 01:28:30.509131 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-snapshots\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.509131 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntm6\" (UniqueName: \"kubernetes.io/projected/53abc737-fbef-4574-96b8-b9adee4fade2-kube-api-access-7ntm6\") pod \"kas-connection-checker-685895c7c9-vg24q\" (UID: \"53abc737-fbef-4574-96b8-b9adee4fade2\") " pod="kube-system/kas-connection-checker-685895c7c9-vg24q" Oct 30 01:28:30.509131 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.509424 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.509424 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.509424 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:30.509424 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8ac7159-dcc1-46dc-bae0-eae434811dbe-serving-cert\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.509424 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78x8b\" (UniqueName: \"kubernetes.io/projected/c36ce70e-acde-4698-89b9-37d85639a669-kube-api-access-78x8b\") pod \"kas-connection-checker-685895c7c9-jlj8q\" (UID: \"c36ce70e-acde-4698-89b9-37d85639a669\") " pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" Oct 30 01:28:30.509424 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad411a5-bb79-4e0c-abd3-85c5ba535c93-tmp-dir\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.509424 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.509424 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aad411a5-bb79-4e0c-abd3-85c5ba535c93-config-volume\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zsmr\" (UniqueName: \"kubernetes.io/projected/3c903b8f-507e-4dec-801e-3213298d7e0a-kube-api-access-7zsmr\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509629 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca1fbc85-1496-4e79-9bc9-330a8540a39e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xfg\" (UniqueName: \"kubernetes.io/projected/b82d0410-941f-43e4-9ae9-5d37f9c409be-kube-api-access-79xfg\") pod \"network-check-source-6f78d59f99-jqlwc\" (UID: \"b82d0410-941f-43e4-9ae9-5d37f9c409be\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx95x\" (UniqueName: \"kubernetes.io/projected/aad411a5-bb79-4e0c-abd3-85c5ba535c93-kube-api-access-bx95x\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a968163-1344-4372-a8b9-bf896d90e6a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqx8q\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.509844 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.510407 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.509901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-config\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.510407 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.510065 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9xkz\" (UniqueName: \"kubernetes.io/projected/6a968163-1344-4372-a8b9-bf896d90e6a9-kube-api-access-x9xkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.510407 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.510083 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.510407 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.510117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c903b8f-507e-4dec-801e-3213298d7e0a-serving-cert\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.510407 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.510183 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:28:30.510407 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.510226 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls podName:bd24e16c-c31b-4af0-b518-cfe8c85937ac nodeName:}" failed. No retries permitted until 2026-10-30 01:28:31.010212019 +0000 UTC m=+31.130859355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5kdp7" (UID: "bd24e16c-c31b-4af0-b518-cfe8c85937ac") : secret "samples-operator-tls" not found Oct 30 01:28:30.510772 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.510497 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a968163-1344-4372-a8b9-bf896d90e6a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.510772 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.510553 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:30.510772 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.510566 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f49dfcbf7-ssq6d: secret "image-registry-tls" not found Oct 30 01:28:30.510772 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.510631 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls podName:7895311d-1a28-4a06-b1b6-2abaa8ed7bbc nodeName:}" failed. No retries permitted until 2026-10-30 01:28:31.010617317 +0000 UTC m=+31.131264654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls") pod "image-registry-7f49dfcbf7-ssq6d" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc") : secret "image-registry-tls" not found Oct 30 01:28:30.510974 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.510951 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aad411a5-bb79-4e0c-abd3-85c5ba535c93-config-volume\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.511313 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.511287 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.511893 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.511871 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:30.512727 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.512353 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-default-certificate\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.512727 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.512406 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:31.012392031 +0000 UTC m=+31.133039369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : configmap references non-existent config key: service-ca.crt Oct 30 01:28:30.512924 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.512831 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.512924 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.512832 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-stats-auth\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.512924 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.511878 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.513435 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.513414 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.513575 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.513561 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:28:30.513662 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.513615 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert podName:bc3b9845-9528-42c3-bdae-dc73fec82643 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:31.013598401 +0000 UTC m=+31.134245722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6sdlf" (UID: "bc3b9845-9528-42c3-bdae-dc73fec82643") : secret "networking-console-plugin-cert" not found Oct 30 01:28:30.513870 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.513851 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:28:30.514026 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.514012 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls podName:aad411a5-bb79-4e0c-abd3-85c5ba535c93 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:31.013995969 +0000 UTC m=+31.134643308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls") pod "dns-default-dvvl4" (UID: "aad411a5-bb79-4e0c-abd3-85c5ba535c93") : secret "dns-default-metrics-tls" not found Oct 30 01:28:30.514129 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.514109 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.514198 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.514093 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8ac7159-dcc1-46dc-bae0-eae434811dbe-config\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.514198 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.514022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c481c3b-feae-4510-83c5-0dc08ce51db0-serving-cert\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.514198 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.513872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.514799 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.514345 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:30.514799 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:30.514506 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls podName:ca1fbc85-1496-4e79-9bc9-330a8540a39e nodeName:}" failed. No retries permitted until 2026-10-30 01:28:31.01449044 +0000 UTC m=+31.135137787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c4msx" (UID: "ca1fbc85-1496-4e79-9bc9-330a8540a39e") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:30.514799 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.514517 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-tmp\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.514799 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.514548 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a968163-1344-4372-a8b9-bf896d90e6a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.514799 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.514667 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-config\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.514799 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.514733 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad411a5-bb79-4e0c-abd3-85c5ba535c93-tmp-dir\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.515199 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.515159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-snapshots\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.515390 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.515373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34fec9a2-a544-4181-bf3a-b3e81ea7929f-cert\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:30.515494 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.515469 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-trusted-ca\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.515669 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.515645 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.515932 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.515909 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8ac7159-dcc1-46dc-bae0-eae434811dbe-serving-cert\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.516364 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.516338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.516541 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.516506 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c903b8f-507e-4dec-801e-3213298d7e0a-serving-cert\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.517246 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.517231 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.550001 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.549976 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8m9r\" (UniqueName: \"kubernetes.io/projected/bd24e16c-c31b-4af0-b518-cfe8c85937ac-kube-api-access-t8m9r\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:30.550187 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.550168 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xfg\" (UniqueName: \"kubernetes.io/projected/b82d0410-941f-43e4-9ae9-5d37f9c409be-kube-api-access-79xfg\") pod \"network-check-source-6f78d59f99-jqlwc\" (UID: \"b82d0410-941f-43e4-9ae9-5d37f9c409be\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" Oct 30 01:28:30.550269 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.550212 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntm6\" (UniqueName: \"kubernetes.io/projected/53abc737-fbef-4574-96b8-b9adee4fade2-kube-api-access-7ntm6\") pod \"kas-connection-checker-685895c7c9-vg24q\" (UID: \"53abc737-fbef-4574-96b8-b9adee4fade2\") " pod="kube-system/kas-connection-checker-685895c7c9-vg24q" Oct 30 01:28:30.550670 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.550655 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx95x\" (UniqueName: \"kubernetes.io/projected/aad411a5-bb79-4e0c-abd3-85c5ba535c93-kube-api-access-bx95x\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:30.551697 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.551677 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp748\" (UniqueName: \"kubernetes.io/projected/a8ac7159-dcc1-46dc-bae0-eae434811dbe-kube-api-access-gp748\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.552034 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.552018 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtlvp\" (UniqueName: \"kubernetes.io/projected/ca1fbc85-1496-4e79-9bc9-330a8540a39e-kube-api-access-rtlvp\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:30.554817 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.554791 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72c7\" (UniqueName: \"kubernetes.io/projected/02ab5bdf-56b9-4176-af0e-f41d92760b84-kube-api-access-f72c7\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:30.555439 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.555422 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gsmz\" (UniqueName: \"kubernetes.io/projected/34fec9a2-a544-4181-bf3a-b3e81ea7929f-kube-api-access-6gsmz\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:30.555735 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.555707 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqx8q\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.555974 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.555956 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9xkz\" (UniqueName: \"kubernetes.io/projected/6a968163-1344-4372-a8b9-bf896d90e6a9-kube-api-access-x9xkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.557694 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.557670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qsv9\" (UniqueName: \"kubernetes.io/projected/2986b483-9c1e-4d59-89d5-01e287807f8a-kube-api-access-6qsv9\") pod \"volume-data-source-validator-58c9f5f46d-srr2t\" (UID: \"2986b483-9c1e-4d59-89d5-01e287807f8a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" Oct 30 01:28:30.559711 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.559691 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fgwl\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.559808 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.559729 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zsmr\" (UniqueName: \"kubernetes.io/projected/3c903b8f-507e-4dec-801e-3213298d7e0a-kube-api-access-7zsmr\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.560436 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.560413 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrh86\" (UniqueName: \"kubernetes.io/projected/7c481c3b-feae-4510-83c5-0dc08ce51db0-kube-api-access-mrh86\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.561363 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.561342 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:30.561923 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.561906 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hb2\" (UniqueName: \"kubernetes.io/projected/eadbf8a4-af76-4fab-b804-13e1ac18afcc-kube-api-access-f2hb2\") pod \"kas-connection-checker-685895c7c9-z42rk\" (UID: \"eadbf8a4-af76-4fab-b804-13e1ac18afcc\") " pod="kube-system/kas-connection-checker-685895c7c9-z42rk" Oct 30 01:28:30.563777 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.563756 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78x8b\" (UniqueName: \"kubernetes.io/projected/c36ce70e-acde-4698-89b9-37d85639a669-kube-api-access-78x8b\") pod \"kas-connection-checker-685895c7c9-jlj8q\" (UID: \"c36ce70e-acde-4698-89b9-37d85639a669\") " pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" Oct 30 01:28:30.564218 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.564198 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:30.570186 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.570169 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z42rk" Oct 30 01:28:30.578034 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.578015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" Oct 30 01:28:30.593521 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.593501 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" Oct 30 01:28:30.612672 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.612641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:30.626936 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.626904 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:30.644302 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.644277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vg24q" Oct 30 01:28:30.651657 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.651631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:30.659955 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.659916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:30.672411 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.672021 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" Oct 30 01:28:30.699549 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.698697 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:30.916471 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.916345 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jlj8q"] Oct 30 01:28:30.919655 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:30.919622 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36ce70e_acde_4698_89b9_37d85639a669.slice/crio-b9efa8322afd0ea236d056ab5cffd195e3d438424e609da773477fccfdce7c05 WatchSource:0}: Error finding container b9efa8322afd0ea236d056ab5cffd195e3d438424e609da773477fccfdce7c05: Status 404 returned error can't find the container with id b9efa8322afd0ea236d056ab5cffd195e3d438424e609da773477fccfdce7c05 Oct 30 01:28:30.920081 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.920059 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t"] Oct 30 01:28:30.923348 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:30.923303 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2986b483_9c1e_4d59_89d5_01e287807f8a.slice/crio-26bd9805c1799a342d75b1d2cb26c7355e0562bbc24b77ffc6da74112c481692 WatchSource:0}: Error finding container 26bd9805c1799a342d75b1d2cb26c7355e0562bbc24b77ffc6da74112c481692: Status 404 returned error can't find the container with id 26bd9805c1799a342d75b1d2cb26c7355e0562bbc24b77ffc6da74112c481692 Oct 30 01:28:30.928171 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.928150 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z42rk"] Oct 30 01:28:30.929475 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:30.929447 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadbf8a4_af76_4fab_b804_13e1ac18afcc.slice/crio-39532e095dc6835edc2016e5c2ac9dae41e2c3b7b23f1ec08983b1231a326ce0 WatchSource:0}: Error finding container 39532e095dc6835edc2016e5c2ac9dae41e2c3b7b23f1ec08983b1231a326ce0: Status 404 returned error can't find the container with id 39532e095dc6835edc2016e5c2ac9dae41e2c3b7b23f1ec08983b1231a326ce0 Oct 30 01:28:30.946417 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.946393 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9659j"] Oct 30 01:28:30.950915 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:30.950883 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c903b8f_507e_4dec_801e_3213298d7e0a.slice/crio-c0db4484c8069d40507da09a5544e6a44216118be03db395b8a3232f20c452af WatchSource:0}: Error finding container c0db4484c8069d40507da09a5544e6a44216118be03db395b8a3232f20c452af: Status 404 returned error can't find the container with id c0db4484c8069d40507da09a5544e6a44216118be03db395b8a3232f20c452af Oct 30 01:28:30.975622 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:30.975596 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rzw29"] Oct 30 01:28:31.010755 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.010731 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h"] Oct 30 01:28:31.013589 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:31.013561 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ac7159_dcc1_46dc_bae0_eae434811dbe.slice/crio-553d0043bae899bc3e7d069be69f093ec235ea2d9ec9086ab8ac2b09ebcb2feb WatchSource:0}: Error finding container 553d0043bae899bc3e7d069be69f093ec235ea2d9ec9086ab8ac2b09ebcb2feb: Status 404 returned error can't find the container with id 553d0043bae899bc3e7d069be69f093ec235ea2d9ec9086ab8ac2b09ebcb2feb Oct 30 01:28:31.020742 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.020716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:31.020819 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.020758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:31.020819 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.020798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:31.020884 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.020830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:31.020884 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.020845 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:28:31.020951 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.020895 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls podName:bd24e16c-c31b-4af0-b518-cfe8c85937ac nodeName:}" failed. No retries permitted until 2026-10-30 01:28:32.020881844 +0000 UTC m=+32.141529182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5kdp7" (UID: "bd24e16c-c31b-4af0-b518-cfe8c85937ac") : secret "samples-operator-tls" not found Oct 30 01:28:31.020951 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.020901 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:28:31.020951 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.020928 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:31.020951 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.020849 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:31.020951 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.020950 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f49dfcbf7-ssq6d: secret "image-registry-tls" not found Oct 30 01:28:31.021100 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.020941 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6976fcd64d-82clx: secret "image-registry-tls" not found Oct 30 01:28:31.021100 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.020950 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls podName:aad411a5-bb79-4e0c-abd3-85c5ba535c93 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:32.020936815 +0000 UTC m=+32.141584135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls") pod "dns-default-dvvl4" (UID: "aad411a5-bb79-4e0c-abd3-85c5ba535c93") : secret "dns-default-metrics-tls" not found Oct 30 01:28:31.021100 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.020981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:31.021100 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.020988 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls podName:8012711c-3e84-43ab-9eac-8e2280709e74 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:32.020979183 +0000 UTC m=+32.141626506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls") pod "image-registry-6976fcd64d-82clx" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74") : secret "image-registry-tls" not found Oct 30 01:28:31.021100 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.021006 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls podName:7895311d-1a28-4a06-b1b6-2abaa8ed7bbc nodeName:}" failed. No retries permitted until 2026-10-30 01:28:32.021000657 +0000 UTC m=+32.141647979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls") pod "image-registry-7f49dfcbf7-ssq6d" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc") : secret "image-registry-tls" not found Oct 30 01:28:31.021100 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.021027 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:28:31.021100 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.021049 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:32.021042244 +0000 UTC m=+32.141689565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : secret "router-metrics-certs-default" not found Oct 30 01:28:31.021100 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.021046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:31.021100 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.021091 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:31.021410 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.021112 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls podName:ca1fbc85-1496-4e79-9bc9-330a8540a39e nodeName:}" failed. No retries permitted until 2026-10-30 01:28:32.021106273 +0000 UTC m=+32.141753593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c4msx" (UID: "ca1fbc85-1496-4e79-9bc9-330a8540a39e") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:31.021410 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.021132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:31.021410 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.021156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:31.021410 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.021215 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:28:31.021410 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.021234 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:32.021221977 +0000 UTC m=+32.141869299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : configmap references non-existent config key: service-ca.crt Oct 30 01:28:31.021410 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:31.021252 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert podName:bc3b9845-9528-42c3-bdae-dc73fec82643 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:32.021245977 +0000 UTC m=+32.141893297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6sdlf" (UID: "bc3b9845-9528-42c3-bdae-dc73fec82643") : secret "networking-console-plugin-cert" not found Oct 30 01:28:31.046511 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.044219 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc"] Oct 30 01:28:31.069285 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.069255 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vg24q"] Oct 30 01:28:31.069329 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.069309 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj"] Oct 30 01:28:31.074229 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.074203 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zmb2x"] Oct 30 01:28:31.074477 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:31.074400 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a968163_1344_4372_a8b9_bf896d90e6a9.slice/crio-e016e044e47012216c57c1498de711374dbfac8918dfbdb2e268162c3f9b5943 WatchSource:0}: Error finding container e016e044e47012216c57c1498de711374dbfac8918dfbdb2e268162c3f9b5943: Status 404 returned error can't find the container with id e016e044e47012216c57c1498de711374dbfac8918dfbdb2e268162c3f9b5943 Oct 30 01:28:31.713116 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.712962 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zmb2x" event={"ID":"34fec9a2-a544-4181-bf3a-b3e81ea7929f","Type":"ContainerStarted","Data":"0caff05880f97cccb5aba759883387b750539454048491c4445102cb3efdcf9a"} Oct 30 01:28:31.718037 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.717962 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" event={"ID":"6a968163-1344-4372-a8b9-bf896d90e6a9","Type":"ContainerStarted","Data":"e016e044e47012216c57c1498de711374dbfac8918dfbdb2e268162c3f9b5943"} Oct 30 01:28:31.719378 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.719350 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9659j" event={"ID":"3c903b8f-507e-4dec-801e-3213298d7e0a","Type":"ContainerStarted","Data":"c0db4484c8069d40507da09a5544e6a44216118be03db395b8a3232f20c452af"} Oct 30 01:28:31.721338 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.721310 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" event={"ID":"c36ce70e-acde-4698-89b9-37d85639a669","Type":"ContainerStarted","Data":"6fb957ec1007bc8d94ef155a6f99618fbe06ce86362a09d040a410899d7f41c6"} Oct 30 01:28:31.721421 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.721354 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" event={"ID":"c36ce70e-acde-4698-89b9-37d85639a669","Type":"ContainerStarted","Data":"b9efa8322afd0ea236d056ab5cffd195e3d438424e609da773477fccfdce7c05"} Oct 30 01:28:31.726266 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.726242 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" event={"ID":"a8ac7159-dcc1-46dc-bae0-eae434811dbe","Type":"ContainerStarted","Data":"553d0043bae899bc3e7d069be69f093ec235ea2d9ec9086ab8ac2b09ebcb2feb"} Oct 30 01:28:31.728519 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.728495 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vg24q" event={"ID":"53abc737-fbef-4574-96b8-b9adee4fade2","Type":"ContainerStarted","Data":"9aa273fae562fccafb9cc69360d12119bc9dcffa8d7e16bc21bc763270064782"} Oct 30 01:28:31.728683 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.728554 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vg24q" event={"ID":"53abc737-fbef-4574-96b8-b9adee4fade2","Type":"ContainerStarted","Data":"3bd27ddb4aadd0f011f371665ead72daa7d96d13bbb970897f7ba7f8e7cc7a61"} Oct 30 01:28:31.732318 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.732293 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" event={"ID":"7c481c3b-feae-4510-83c5-0dc08ce51db0","Type":"ContainerStarted","Data":"6c3e87c3c22046833f4b5eeafcdf36c1842b846590bc4a8f0fbf0dfa300d30c5"} Oct 30 01:28:31.734375 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.734350 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z42rk" event={"ID":"eadbf8a4-af76-4fab-b804-13e1ac18afcc","Type":"ContainerStarted","Data":"1259b5f4cef9befa0dbb12d7950adf9ab4a88572908520cb93755b722c80e440"} Oct 30 01:28:31.734457 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.734392 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z42rk" event={"ID":"eadbf8a4-af76-4fab-b804-13e1ac18afcc","Type":"ContainerStarted","Data":"39532e095dc6835edc2016e5c2ac9dae41e2c3b7b23f1ec08983b1231a326ce0"} Oct 30 01:28:31.737838 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.737757 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" event={"ID":"b82d0410-941f-43e4-9ae9-5d37f9c409be","Type":"ContainerStarted","Data":"54475ade28d87774d9457a4878e1f2f8aa10364657f4ec57c9c6a2e3ffbee909"} Oct 30 01:28:31.740262 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.740213 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" event={"ID":"2986b483-9c1e-4d59-89d5-01e287807f8a","Type":"ContainerStarted","Data":"26bd9805c1799a342d75b1d2cb26c7355e0562bbc24b77ffc6da74112c481692"} Oct 30 01:28:31.769014 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.768952 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" podStartSLOduration=369.768935005 podStartE2EDuration="6m9.768935005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:31.766826705 +0000 UTC m=+31.887474040" watchObservedRunningTime="2026-10-30 01:28:31.768935005 +0000 UTC m=+31.889582349" Oct 30 01:28:31.867982 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:31.867925 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-z42rk" podStartSLOduration=370.867906325 podStartE2EDuration="6m10.867906325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:31.814831693 +0000 UTC m=+31.935479037" watchObservedRunningTime="2026-10-30 01:28:31.867906325 +0000 UTC m=+31.988553670" Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:32.028911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:32.028979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:32.029054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029059 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:32.029084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029125 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.02910738 +0000 UTC m=+34.149754718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : secret "router-metrics-certs-default" not found Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029163 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029200 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert podName:bc3b9845-9528-42c3-bdae-dc73fec82643 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.029187533 +0000 UTC m=+34.149834870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6sdlf" (UID: "bc3b9845-9528-42c3-bdae-dc73fec82643") : secret "networking-console-plugin-cert" not found Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:32.029160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029213 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:32.029228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029244 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls podName:bd24e16c-c31b-4af0-b518-cfe8c85937ac nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.029234162 +0000 UTC m=+34.149881500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5kdp7" (UID: "bd24e16c-c31b-4af0-b518-cfe8c85937ac") : secret "samples-operator-tls" not found Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:32.029272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029286 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029297 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f49dfcbf7-ssq6d: secret "image-registry-tls" not found Oct 30 01:28:32.029686 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:32.029307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:32.030557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029329 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls podName:7895311d-1a28-4a06-b1b6-2abaa8ed7bbc nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.029319704 +0000 UTC m=+34.149967026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls") pod "image-registry-7f49dfcbf7-ssq6d" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc") : secret "image-registry-tls" not found Oct 30 01:28:32.030557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029363 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:32.030557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029373 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6976fcd64d-82clx: secret "image-registry-tls" not found Oct 30 01:28:32.030557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029401 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls podName:8012711c-3e84-43ab-9eac-8e2280709e74 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.02939245 +0000 UTC m=+34.150039775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls") pod "image-registry-6976fcd64d-82clx" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74") : secret "image-registry-tls" not found Oct 30 01:28:32.030557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029426 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:32.030557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029456 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls podName:ca1fbc85-1496-4e79-9bc9-330a8540a39e nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.02944673 +0000 UTC m=+34.150094056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c4msx" (UID: "ca1fbc85-1496-4e79-9bc9-330a8540a39e") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:32.030557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029512 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.029502647 +0000 UTC m=+34.150149969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : configmap references non-existent config key: service-ca.crt Oct 30 01:28:32.030557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029585 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:28:32.030557 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:32.029617 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls podName:aad411a5-bb79-4e0c-abd3-85c5ba535c93 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:34.029606371 +0000 UTC m=+34.150253707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls") pod "dns-default-dvvl4" (UID: "aad411a5-bb79-4e0c-abd3-85c5ba535c93") : secret "dns-default-metrics-tls" not found Oct 30 01:28:33.751161 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:33.751125 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" event={"ID":"2986b483-9c1e-4d59-89d5-01e287807f8a","Type":"ContainerStarted","Data":"8b8da04d1c5edcb07b185a4aaf93eedd30589a740790e6550ebf0ede6ddad24c"} Oct 30 01:28:33.796298 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:33.796243 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vg24q" podStartSLOduration=371.796226812 podStartE2EDuration="6m11.796226812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:31.86827005 +0000 UTC m=+31.988917393" watchObservedRunningTime="2026-10-30 01:28:33.796226812 +0000 UTC m=+33.916874161" Oct 30 01:28:33.796578 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:33.796542 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" podStartSLOduration=342.715522764 podStartE2EDuration="5m44.796519164s" totalImagesPullingTime="2.0809964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:33.79572433 +0000 UTC m=+33.916371675" watchObservedRunningTime="2026-10-30 01:28:33.796519164 +0000 UTC m=+33.917166509" Oct 30 01:28:34.049048 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.048945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:34.049048 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.049003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:34.049048 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.049034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.049069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049125 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049171 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049193 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f49dfcbf7-ssq6d: secret "image-registry-tls" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049205 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls podName:bd24e16c-c31b-4af0-b518-cfe8c85937ac nodeName:}" failed. No retries permitted until 2026-10-30 01:28:38.049185462 +0000 UTC m=+38.169832788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5kdp7" (UID: "bd24e16c-c31b-4af0-b518-cfe8c85937ac") : secret "samples-operator-tls" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049211 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049247 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls podName:7895311d-1a28-4a06-b1b6-2abaa8ed7bbc nodeName:}" failed. No retries permitted until 2026-10-30 01:28:38.049229281 +0000 UTC m=+38.169876621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls") pod "image-registry-7f49dfcbf7-ssq6d" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc") : secret "image-registry-tls" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049270 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:38.049259931 +0000 UTC m=+38.169907257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : secret "router-metrics-certs-default" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049291 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049327 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls podName:aad411a5-bb79-4e0c-abd3-85c5ba535c93 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:38.049313057 +0000 UTC m=+38.169960393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls") pod "dns-default-dvvl4" (UID: "aad411a5-bb79-4e0c-abd3-85c5ba535c93") : secret "dns-default-metrics-tls" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.049134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049339 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:34.049348 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049352 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6976fcd64d-82clx: secret "image-registry-tls" not found Oct 30 01:28:34.050028 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049388 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls podName:8012711c-3e84-43ab-9eac-8e2280709e74 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:38.049375643 +0000 UTC m=+38.170023002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls") pod "image-registry-6976fcd64d-82clx" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74") : secret "image-registry-tls" not found Oct 30 01:28:34.050028 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.049383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:34.050028 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049439 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:34.050028 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049473 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls podName:ca1fbc85-1496-4e79-9bc9-330a8540a39e nodeName:}" failed. No retries permitted until 2026-10-30 01:28:38.049461483 +0000 UTC m=+38.170108808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c4msx" (UID: "ca1fbc85-1496-4e79-9bc9-330a8540a39e") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:34.050028 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.049491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:34.050028 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.049522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:34.050028 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049631 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:28:34.050028 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049650 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:38.049634497 +0000 UTC m=+38.170281833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : configmap references non-existent config key: service-ca.crt Oct 30 01:28:34.050028 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.049671 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert podName:bc3b9845-9528-42c3-bdae-dc73fec82643 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:38.049662232 +0000 UTC m=+38.170309557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6sdlf" (UID: "bc3b9845-9528-42c3-bdae-dc73fec82643") : secret "networking-console-plugin-cert" not found Oct 30 01:28:34.150169 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.150137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:34.150343 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.150181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:34.150476 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.150457 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 01:28:34.150598 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:34.150518 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:29:06.15050236 +0000 UTC m=+66.271149681 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : secret "metrics-daemon-secret" not found Oct 30 01:28:34.156517 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.156483 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:34.251002 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.250968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:34.253741 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.253716 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:34.363087 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.363010 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:34.370173 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.370143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:34.763541 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.763500 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/0.log" Oct 30 01:28:34.764100 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.764073 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c481c3b-feae-4510-83c5-0dc08ce51db0" containerID="6bb6667db0ff1eb8ac841148ab95785eb4b80c3aed89cce0be9225874f69d2df" exitCode=255 Oct 30 01:28:34.764687 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.764247 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" event={"ID":"7c481c3b-feae-4510-83c5-0dc08ce51db0","Type":"ContainerDied","Data":"6bb6667db0ff1eb8ac841148ab95785eb4b80c3aed89cce0be9225874f69d2df"} Oct 30 01:28:34.764687 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.764464 2576 scope.go:122] "RemoveContainer" containerID="6bb6667db0ff1eb8ac841148ab95785eb4b80c3aed89cce0be9225874f69d2df" Oct 30 01:28:34.778467 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.778442 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hsc2x"] Oct 30 01:28:34.816186 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:34.816161 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xknfd"] Oct 30 01:28:34.826666 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:34.826642 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fed544_0d4f_4a7b_9ba6_1b47bb5eaa92.slice/crio-e93c31328aadee075019623b5e993e1754e19f391920cbf8d666f671b7ef129b WatchSource:0}: Error finding container e93c31328aadee075019623b5e993e1754e19f391920cbf8d666f671b7ef129b: Status 404 returned error can't find the container with id e93c31328aadee075019623b5e993e1754e19f391920cbf8d666f671b7ef129b Oct 30 01:28:35.772240 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.772198 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xknfd" event={"ID":"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92","Type":"ContainerStarted","Data":"e93c31328aadee075019623b5e993e1754e19f391920cbf8d666f671b7ef129b"} Oct 30 01:28:35.778618 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.777889 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" event={"ID":"6a968163-1344-4372-a8b9-bf896d90e6a9","Type":"ContainerStarted","Data":"347ea1d8a0df4785c02866a1aa271554539f3f8cdaf782bdfed1304ad273c911"} Oct 30 01:28:35.780555 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.780409 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9659j" event={"ID":"3c903b8f-507e-4dec-801e-3213298d7e0a","Type":"ContainerStarted","Data":"3a10c0fd672b6ab8b07d93f7b29c15ce7079b0d7af9f13bbc1eaf4d9662ede86"} Oct 30 01:28:35.782230 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.782204 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hsc2x" event={"ID":"971584c9-502d-4dfa-beae-8d8d7ae88cb3","Type":"ContainerStarted","Data":"df1689303ba6d4314fe08e797680b367740df33fd1d7da9f114eb0ff6312e878"} Oct 30 01:28:35.786378 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.786360 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/1.log" Oct 30 01:28:35.787639 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.786848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/0.log" Oct 30 01:28:35.787639 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.786885 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c481c3b-feae-4510-83c5-0dc08ce51db0" containerID="d3cbef498b1844773837fd0509bdb11f4bc072e0f4f21fe5b4a4fbd3171d9e31" exitCode=255 Oct 30 01:28:35.787639 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.786917 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" event={"ID":"7c481c3b-feae-4510-83c5-0dc08ce51db0","Type":"ContainerDied","Data":"d3cbef498b1844773837fd0509bdb11f4bc072e0f4f21fe5b4a4fbd3171d9e31"} Oct 30 01:28:35.787639 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.786962 2576 scope.go:122] "RemoveContainer" containerID="6bb6667db0ff1eb8ac841148ab95785eb4b80c3aed89cce0be9225874f69d2df" Oct 30 01:28:35.787639 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.787283 2576 scope.go:122] "RemoveContainer" containerID="d3cbef498b1844773837fd0509bdb11f4bc072e0f4f21fe5b4a4fbd3171d9e31" Oct 30 01:28:35.787639 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:35.787418 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rzw29_openshift-console-operator(7c481c3b-feae-4510-83c5-0dc08ce51db0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" podUID="7c481c3b-feae-4510-83c5-0dc08ce51db0" Oct 30 01:28:35.845267 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.845143 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" podStartSLOduration=341.575254518 podStartE2EDuration="5m45.845123476s" totalImagesPullingTime="4.269868958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:35.842185261 +0000 UTC m=+35.962832603" watchObservedRunningTime="2026-10-30 01:28:35.845123476 +0000 UTC m=+35.965770823" Oct 30 01:28:35.888181 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.884627 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hflnn"] Oct 30 01:28:35.894569 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.891907 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-9659j" podStartSLOduration=342.8523123 podStartE2EDuration="5m46.891889036s" totalImagesPullingTime="4.039576736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:35.890374722 +0000 UTC m=+36.011022067" watchObservedRunningTime="2026-10-30 01:28:35.891889036 +0000 UTC m=+36.012536379" Oct 30 01:28:35.916869 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.916841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:35.923759 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.923731 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hflnn"] Oct 30 01:28:35.935547 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.935240 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:28:35.935748 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.935702 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfcpg\"" Oct 30 01:28:35.936406 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.936375 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:28:35.971116 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.971001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hrqt\" (UniqueName: \"kubernetes.io/projected/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-api-access-4hrqt\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:35.971277 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.971178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10d07cdc-86b5-4181-9b95-9cfb0341db09-data-volume\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:35.971475 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.971450 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vvvgp-9njqn"] Oct 30 01:28:35.971556 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.971449 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:35.971613 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.971592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:35.971669 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.971651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10d07cdc-86b5-4181-9b95-9cfb0341db09-crio-socket\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:35.989287 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:35.989228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.002090 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.002063 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vvvgp-9njqn"] Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.072638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.072678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.072710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5xd\" (UniqueName: \"kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.072741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.072785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10d07cdc-86b5-4181-9b95-9cfb0341db09-crio-socket\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.072810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.072883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrqt\" (UniqueName: \"kubernetes.io/projected/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-api-access-4hrqt\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.072978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.073020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10d07cdc-86b5-4181-9b95-9cfb0341db09-data-volume\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.073363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10d07cdc-86b5-4181-9b95-9cfb0341db09-data-volume\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:36.073438 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:36.073491 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:36.573472212 +0000 UTC m=+36.694119549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : configmap "kube-rbac-proxy" not found Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:36.073771 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:36.073816 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:36.573802004 +0000 UTC m=+36.694449340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : secret "insights-runtime-extractor-tls" not found Oct 30 01:28:36.091004 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.073910 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10d07cdc-86b5-4181-9b95-9cfb0341db09-crio-socket\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.099586 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.098926 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrqt\" (UniqueName: \"kubernetes.io/projected/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-api-access-4hrqt\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.174097 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.174062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.174240 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.174222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.174365 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.174349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5xd\" (UniqueName: \"kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.174416 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.174387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.174454 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.174439 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.175095 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.175071 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.175221 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.175203 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.200610 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.200561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5xd\" (UniqueName: \"kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.325114 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.325037 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:36.566032 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.565997 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c"] Oct 30 01:28:36.578757 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.578652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.578757 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.578691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:36.579065 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:36.578906 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 01:28:36.579065 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:36.578951 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:37.578939889 +0000 UTC m=+37.699587210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : configmap "kube-rbac-proxy" not found Oct 30 01:28:36.579065 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:36.578999 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:28:36.579065 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:36.579020 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:37.579013888 +0000 UTC m=+37.699661209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : secret "insights-runtime-extractor-tls" not found Oct 30 01:28:36.584566 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.584220 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4"] Oct 30 01:28:36.587960 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.584726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" Oct 30 01:28:36.600274 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.600255 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c"] Oct 30 01:28:36.600357 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.600286 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4"] Oct 30 01:28:36.600357 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.600354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" Oct 30 01:28:36.610682 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.609544 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:36.610682 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.610235 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sj9w4\"" Oct 30 01:28:36.610682 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.610320 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 01:28:36.680229 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.680194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hrx\" (UniqueName: \"kubernetes.io/projected/32c651f8-7083-4335-a937-682d7a665312-kube-api-access-97hrx\") pod \"migrator-66df5d496b-fdcg4\" (UID: \"32c651f8-7083-4335-a937-682d7a665312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" Oct 30 01:28:36.680396 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.680288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hmfr\" (UniqueName: \"kubernetes.io/projected/1989a7c0-f591-413a-b111-266ad115c63a-kube-api-access-8hmfr\") pod \"migrator-66df5d496b-5jk7c\" (UID: \"1989a7c0-f591-413a-b111-266ad115c63a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" Oct 30 01:28:36.722607 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.722575 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vvvgp-9njqn"] Oct 30 01:28:36.724874 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:36.724843 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118aebf2_95cc_4c0a_8f6d_fed0c906316d.slice/crio-d12efdef0c32486f8df86324b241438773d6318337ca920f2cc7b474630fe4b5 WatchSource:0}: Error finding container d12efdef0c32486f8df86324b241438773d6318337ca920f2cc7b474630fe4b5: Status 404 returned error can't find the container with id d12efdef0c32486f8df86324b241438773d6318337ca920f2cc7b474630fe4b5 Oct 30 01:28:36.781809 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.781778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmfr\" (UniqueName: \"kubernetes.io/projected/1989a7c0-f591-413a-b111-266ad115c63a-kube-api-access-8hmfr\") pod \"migrator-66df5d496b-5jk7c\" (UID: \"1989a7c0-f591-413a-b111-266ad115c63a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" Oct 30 01:28:36.782657 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.782073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hrx\" (UniqueName: \"kubernetes.io/projected/32c651f8-7083-4335-a937-682d7a665312-kube-api-access-97hrx\") pod \"migrator-66df5d496b-fdcg4\" (UID: \"32c651f8-7083-4335-a937-682d7a665312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" Oct 30 01:28:36.799908 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.799246 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" event={"ID":"a8ac7159-dcc1-46dc-bae0-eae434811dbe","Type":"ContainerStarted","Data":"3a9844d501e478e73a0af99607fb9948506f820dde7ea2400d2c47d9c90d6d22"} Oct 30 01:28:36.806282 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.806260 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/1.log" Oct 30 01:28:36.807185 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.806668 2576 scope.go:122] "RemoveContainer" containerID="d3cbef498b1844773837fd0509bdb11f4bc072e0f4f21fe5b4a4fbd3171d9e31" Oct 30 01:28:36.807185 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:36.806856 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rzw29_openshift-console-operator(7c481c3b-feae-4510-83c5-0dc08ce51db0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" podUID="7c481c3b-feae-4510-83c5-0dc08ce51db0" Oct 30 01:28:36.810663 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.810632 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" event={"ID":"118aebf2-95cc-4c0a-8f6d-fed0c906316d","Type":"ContainerStarted","Data":"d12efdef0c32486f8df86324b241438773d6318337ca920f2cc7b474630fe4b5"} Oct 30 01:28:36.821804 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.821780 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hrx\" (UniqueName: \"kubernetes.io/projected/32c651f8-7083-4335-a937-682d7a665312-kube-api-access-97hrx\") pod \"migrator-66df5d496b-fdcg4\" (UID: \"32c651f8-7083-4335-a937-682d7a665312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" Oct 30 01:28:36.824464 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.824437 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zmb2x" event={"ID":"34fec9a2-a544-4181-bf3a-b3e81ea7929f","Type":"ContainerStarted","Data":"492085e224736ecaddcdf5d47fed9baff5d0ca5dbbc6f54b61183459413558fb"} Oct 30 01:28:36.829610 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.829561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmfr\" (UniqueName: \"kubernetes.io/projected/1989a7c0-f591-413a-b111-266ad115c63a-kube-api-access-8hmfr\") pod \"migrator-66df5d496b-5jk7c\" (UID: \"1989a7c0-f591-413a-b111-266ad115c63a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" Oct 30 01:28:36.892995 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.892949 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" podStartSLOduration=343.169140575 podStartE2EDuration="5m47.892932179s" totalImagesPullingTime="4.723791604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:36.847393101 +0000 UTC m=+36.968040445" watchObservedRunningTime="2026-10-30 01:28:36.892932179 +0000 UTC m=+37.013579540" Oct 30 01:28:36.893183 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.893038 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zmb2x" podStartSLOduration=1.534612939 podStartE2EDuration="6.893033232s" totalImagesPullingTime="5.358420293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:36.89247475 +0000 UTC m=+37.013122084" watchObservedRunningTime="2026-10-30 01:28:36.893033232 +0000 UTC m=+37.013680576" Oct 30 01:28:36.923043 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.922975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" Oct 30 01:28:36.945336 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:36.945272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" Oct 30 01:28:37.240459 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.240433 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c"] Oct 30 01:28:37.252559 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.251608 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-b9g78"] Oct 30 01:28:37.272455 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:37.272411 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c651f8_7083_4335_a937_682d7a665312.slice/crio-e964e4d9a4ea604160731a35bc55ad8b8d907ce3c62b8b19c1ad2268bf355772 WatchSource:0}: Error finding container e964e4d9a4ea604160731a35bc55ad8b8d907ce3c62b8b19c1ad2268bf355772: Status 404 returned error can't find the container with id e964e4d9a4ea604160731a35bc55ad8b8d907ce3c62b8b19c1ad2268bf355772 Oct 30 01:28:37.288920 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.288861 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4"] Oct 30 01:28:37.289446 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.289170 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.290124 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.289977 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-b9g78"] Oct 30 01:28:37.307195 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.307074 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 01:28:37.307195 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.307170 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9qrd6\"" Oct 30 01:28:37.307470 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.307448 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 01:28:37.313096 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.313006 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 01:28:37.313096 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.313065 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 01:28:37.392763 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.392732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bdfx\" (UniqueName: \"kubernetes.io/projected/b011ba7c-f34c-44e2-9498-5bd998c41b07-kube-api-access-9bdfx\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.392920 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.392811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-cabundle\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.392920 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.392841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-key\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.494273 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.494197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-cabundle\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.494273 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.494235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-key\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.494487 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.494389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bdfx\" (UniqueName: \"kubernetes.io/projected/b011ba7c-f34c-44e2-9498-5bd998c41b07-kube-api-access-9bdfx\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.495097 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.495066 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-cabundle\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.497661 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.497638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-key\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.520522 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.520475 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bdfx\" (UniqueName: \"kubernetes.io/projected/b011ba7c-f34c-44e2-9498-5bd998c41b07-kube-api-access-9bdfx\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.595234 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.595041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:37.595234 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.595088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:37.595234 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:37.595165 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 01:28:37.595234 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:37.595230 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:39.59521396 +0000 UTC m=+39.715861302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : configmap "kube-rbac-proxy" not found Oct 30 01:28:37.595523 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:37.595277 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:28:37.595523 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:37.595346 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:39.595326291 +0000 UTC m=+39.715973629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : secret "insights-runtime-extractor-tls" not found Oct 30 01:28:37.624544 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.624503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:37.831871 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.831776 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" event={"ID":"118aebf2-95cc-4c0a-8f6d-fed0c906316d","Type":"ContainerStarted","Data":"49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb"} Oct 30 01:28:37.834470 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.834438 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" event={"ID":"32c651f8-7083-4335-a937-682d7a665312","Type":"ContainerStarted","Data":"e964e4d9a4ea604160731a35bc55ad8b8d907ce3c62b8b19c1ad2268bf355772"} Oct 30 01:28:37.836293 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.836266 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" event={"ID":"1989a7c0-f591-413a-b111-266ad115c63a","Type":"ContainerStarted","Data":"3d48388582a3edbde0b946e0c03f5e00d19b80127b216772a8a11eef22beba85"} Oct 30 01:28:37.838124 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.838066 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" event={"ID":"b82d0410-941f-43e4-9ae9-5d37f9c409be","Type":"ContainerStarted","Data":"adf250185eac31386c24ddc8513ffe7df4b154d9a4f64157212de4ed820570af"} Oct 30 01:28:37.878386 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.878332 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" podStartSLOduration=2.8783156229999998 podStartE2EDuration="2.878315623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:37.877168879 +0000 UTC m=+37.997816222" watchObservedRunningTime="2026-10-30 01:28:37.878315623 +0000 UTC m=+37.998962967" Oct 30 01:28:37.922846 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.922792 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" podStartSLOduration=334.771417753 podStartE2EDuration="5m40.922772843s" totalImagesPullingTime="6.15135509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:37.921585043 +0000 UTC m=+38.042232386" watchObservedRunningTime="2026-10-30 01:28:37.922772843 +0000 UTC m=+38.043420187" Oct 30 01:28:37.950616 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:37.950336 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-b9g78"] Oct 30 01:28:37.953487 ip-10-0-142-110 kubenswrapper[2576]: W1030 01:28:37.953455 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb011ba7c_f34c_44e2_9498_5bd998c41b07.slice/crio-34b6335dff88b63b558f7923f9ba4b65fec5e0e761f669a0786ebdaf281e4896 WatchSource:0}: Error finding container 34b6335dff88b63b558f7923f9ba4b65fec5e0e761f669a0786ebdaf281e4896: Status 404 returned error can't find the container with id 34b6335dff88b63b558f7923f9ba4b65fec5e0e761f669a0786ebdaf281e4896 Oct 30 01:28:38.100356 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.100315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:38.100436 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100419 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:38.100484 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100441 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6976fcd64d-82clx: secret "image-registry-tls" not found Oct 30 01:28:38.100567 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100500 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls podName:8012711c-3e84-43ab-9eac-8e2280709e74 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.100481338 +0000 UTC m=+46.221128661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls") pod "image-registry-6976fcd64d-82clx" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74") : secret "image-registry-tls" not found Oct 30 01:28:38.100567 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.100438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:38.100567 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100541 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:28:38.100724 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.100579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:38.100724 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.100580157 +0000 UTC m=+46.221227493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : secret "router-metrics-certs-default" not found Oct 30 01:28:38.100724 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100654 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:38.100724 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.100677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:38.100724 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100692 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls podName:ca1fbc85-1496-4e79-9bc9-330a8540a39e nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.100679125 +0000 UTC m=+46.221326446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c4msx" (UID: "ca1fbc85-1496-4e79-9bc9-330a8540a39e") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.100726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100753 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.100741987 +0000 UTC m=+46.221389325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : configmap references non-existent config key: service-ca.crt Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100798 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.100804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100832 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert podName:bc3b9845-9528-42c3-bdae-dc73fec82643 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.100819706 +0000 UTC m=+46.221467029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6sdlf" (UID: "bc3b9845-9528-42c3-bdae-dc73fec82643") : secret "networking-console-plugin-cert" not found Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.100853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100876 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.100888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100912 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls podName:bd24e16c-c31b-4af0-b518-cfe8c85937ac nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.100901614 +0000 UTC m=+46.221548967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5kdp7" (UID: "bd24e16c-c31b-4af0-b518-cfe8c85937ac") : secret "samples-operator-tls" not found Oct 30 01:28:38.100969 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100951 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:28:38.101323 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.100983 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls podName:aad411a5-bb79-4e0c-abd3-85c5ba535c93 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.100972027 +0000 UTC m=+46.221619351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls") pod "dns-default-dvvl4" (UID: "aad411a5-bb79-4e0c-abd3-85c5ba535c93") : secret "dns-default-metrics-tls" not found Oct 30 01:28:38.101323 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.101018 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:38.101323 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.101030 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f49dfcbf7-ssq6d: secret "image-registry-tls" not found Oct 30 01:28:38.101323 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:38.101062 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls podName:7895311d-1a28-4a06-b1b6-2abaa8ed7bbc nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.101051318 +0000 UTC m=+46.221698659 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls") pod "image-registry-7f49dfcbf7-ssq6d" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc") : secret "image-registry-tls" not found Oct 30 01:28:38.844020 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.843983 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hsc2x" event={"ID":"971584c9-502d-4dfa-beae-8d8d7ae88cb3","Type":"ContainerStarted","Data":"0d1ba8b425b92e79111b39e4e0568bde96fb37a4d51f6dcd683119e0dd0a4878"} Oct 30 01:28:38.844472 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.844103 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:38.846087 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.846060 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" event={"ID":"b011ba7c-f34c-44e2-9498-5bd998c41b07","Type":"ContainerStarted","Data":"10e3c390d41c2c0ae9b0451acfaf6df59c4a5b857e0676aae48bfe3c5b3b7a36"} Oct 30 01:28:38.846205 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.846106 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" event={"ID":"b011ba7c-f34c-44e2-9498-5bd998c41b07","Type":"ContainerStarted","Data":"34b6335dff88b63b558f7923f9ba4b65fec5e0e761f669a0786ebdaf281e4896"} Oct 30 01:28:38.896498 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.896443 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hsc2x" podStartSLOduration=34.604573726 podStartE2EDuration="37.896425317s" totalImagesPullingTime="3.291851591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:38.889872457 +0000 UTC m=+39.010519811" watchObservedRunningTime="2026-10-30 01:28:38.896425317 +0000 UTC m=+39.017072660" Oct 30 01:28:38.936831 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:38.936788 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" podStartSLOduration=1.936774105 podStartE2EDuration="1.936774105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:38.93567498 +0000 UTC m=+39.056322327" watchObservedRunningTime="2026-10-30 01:28:38.936774105 +0000 UTC m=+39.057421450" Oct 30 01:28:39.414350 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:39.414197 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bv8wk_22ff8c4e-b07d-417b-b4d3-61560b014b77/dns-node-resolver/0.log" Oct 30 01:28:39.619950 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:39.619914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:39.619950 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:39.619950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:39.620151 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:39.620040 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 01:28:39.620151 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:39.620115 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:43.620094292 +0000 UTC m=+43.740741629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : configmap "kube-rbac-proxy" not found Oct 30 01:28:39.620151 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:39.620136 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:28:39.620306 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:39.620177 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:43.620166039 +0000 UTC m=+43.740813373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : secret "insights-runtime-extractor-tls" not found Oct 30 01:28:39.851472 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:39.851433 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" event={"ID":"32c651f8-7083-4335-a937-682d7a665312","Type":"ContainerStarted","Data":"f4872fd87e29dfc1f3c054167278e59ecfbd83b6baa7ea3cde91e6ccdab71959"} Oct 30 01:28:39.853077 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:39.853048 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" event={"ID":"1989a7c0-f591-413a-b111-266ad115c63a","Type":"ContainerStarted","Data":"4b63d39670bb5db49d45091b1762d9b5e5161583feb5f31b53cdd65076487e64"} Oct 30 01:28:39.896056 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:39.896002 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" podStartSLOduration=2.063595634 podStartE2EDuration="3.895984863s" totalImagesPullingTime="1.832389229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:39.89478997 +0000 UTC m=+40.015437313" watchObservedRunningTime="2026-10-30 01:28:39.895984863 +0000 UTC m=+40.016632243" Oct 30 01:28:39.938308 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:39.938260 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" podStartSLOduration=2.076362808 podStartE2EDuration="3.93824627s" totalImagesPullingTime="1.861883462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:39.937886221 +0000 UTC m=+40.058533563" watchObservedRunningTime="2026-10-30 01:28:39.93824627 +0000 UTC m=+40.058893616" Oct 30 01:28:40.012397 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:40.012369 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8kspl_acf8cbe6-582e-408b-85a9-d969e18b4682/node-ca/0.log" Oct 30 01:28:40.612791 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:40.612754 2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:40.612973 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:40.612809 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:40.613200 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:40.613181 2576 scope.go:122] "RemoveContainer" containerID="d3cbef498b1844773837fd0509bdb11f4bc072e0f4f21fe5b4a4fbd3171d9e31" Oct 30 01:28:40.613402 ip-10-0-142-110 kubenswrapper[2576]: E1030 01:28:40.613383 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rzw29_openshift-console-operator(7c481c3b-feae-4510-83c5-0dc08ce51db0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" podUID="7c481c3b-feae-4510-83c5-0dc08ce51db0" Oct 30 01:28:40.811649 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:40.811617 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zmb2x_34fec9a2-a544-4181-bf3a-b3e81ea7929f/serve-healthcheck-canary/0.log" Oct 30 01:28:41.012063 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:41.012034 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5jk7c_1989a7c0-f591-413a-b111-266ad115c63a/migrator/0.log" Oct 30 01:28:41.211981 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:41.211961 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fdcg4_32c651f8-7083-4335-a937-682d7a665312/migrator/0.log" Oct 30 01:28:41.413493 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:41.413419 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj_6a968163-1344-4372-a8b9-bf896d90e6a9/kube-storage-version-migrator-operator/0.log" Oct 30 01:28:41.447561 ip-10-0-142-110 kubenswrapper[2576]: I1030 01:28:41.446111 2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:28:41.446323 ip-10-0-142-110 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 01:28:41.455838 ip-10-0-142-110 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 01:28:41.456009 ip-10-0-142-110 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 01:28:41.456055 ip-10-0-142-110 systemd[1]: kubelet.service: Consumed 3.127s CPU time. Oct 30 01:28:41.467157 ip-10-0-142-110 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:28:41.544014 ip-10-0-142-110 kubenswrapper[5035]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:41.544014 ip-10-0-142-110 kubenswrapper[5035]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:28:41.544014 ip-10-0-142-110 kubenswrapper[5035]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:41.544014 ip-10-0-142-110 kubenswrapper[5035]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:41.546279 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546194 5035 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:41.546279 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546274 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:41.546279 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546279 5035 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:41.546279 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546283 5035 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546287 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546292 5035 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546296 5035 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546301 5035 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546304 5035 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546308 5035 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546312 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546316 5035 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546320 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546324 5035 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546328 5035 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546341 5035 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546346 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546350 5035 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546355 5035 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546359 5035 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546364 5035 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546368 5035 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546372 5035 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:41.546559 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546376 5035 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546380 5035 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546385 5035 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546389 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546393 5035 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546397 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546401 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546406 5035 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546410 5035 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546415 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546419 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546423 5035 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546428 5035 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546432 5035 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546437 5035 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546441 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546446 5035 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546450 5035 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546455 5035 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546470 5035 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:41.547421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546475 5035 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546479 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546484 5035 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546488 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546493 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546497 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546504 5035 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546508 5035 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546512 5035 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546517 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546539 5035 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546547 5035 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546553 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546557 5035 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546562 5035 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546566 5035 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546570 5035 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546575 5035 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546579 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:41.548315 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546584 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546589 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546593 5035 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546597 5035 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546601 5035 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546605 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546610 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546615 5035 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546619 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546623 5035 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546628 5035 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546632 5035 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546636 5035 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546652 5035 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546659 5035 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546665 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546670 5035 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546677 5035 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546682 5035 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:41.549178 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546687 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546691 5035 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546696 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546701 5035 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546706 5035 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546710 5035 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546714 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546718 5035 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546723 5035 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546727 5035 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546731 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546735 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546740 5035 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546744 5035 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546748 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546753 5035 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546757 5035 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546761 5035 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546765 5035 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:41.550042 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546772 5035 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546777 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546781 5035 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546786 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546790 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.546794 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548290 5035 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548304 5035 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548312 5035 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548317 5035 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548321 5035 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548326 5035 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548330 5035 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548334 5035 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548339 5035 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548345 5035 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548352 5035 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548357 5035 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548361 5035 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:41.550902 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548366 5035 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548370 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548374 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548379 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548383 5035 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548387 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548392 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548396 5035 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548401 5035 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548406 5035 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548410 5035 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548414 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548420 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548425 5035 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548429 5035 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548433 5035 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548438 5035 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548442 5035 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548447 5035 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548451 5035 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:41.551800 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548456 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548460 5035 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548465 5035 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548470 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548474 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548478 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548482 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548487 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548491 5035 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548495 5035 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548500 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548504 5035 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548508 5035 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548512 5035 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548518 5035 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548522 5035 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548543 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548547 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548552 5035 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:41.552696 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548556 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548560 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548565 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548569 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548574 5035 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548578 5035 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548582 5035 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548587 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548591 5035 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548595 5035 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548599 5035 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548604 5035 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548608 5035 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548612 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548617 5035 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548621 5035 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548627 5035 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548631 5035 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548635 5035 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548639 5035 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:41.553271 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548644 5035 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548648 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548652 5035 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548657 5035 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548661 5035 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548666 5035 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548671 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548675 5035 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548679 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548683 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548688 5035 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548692 5035 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548696 5035 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548701 5035 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548708 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548712 5035 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548717 5035 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548721 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548725 5035 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548729 5035 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:41.553773 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548734 5035 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548738 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548742 5035 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548747 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548751 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548755 5035 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548760 5035 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548764 5035 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548768 5035 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548780 5035 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548785 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548789 5035 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548794 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.548798 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549580 5035 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549590 5035 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549594 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549601 5035 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549606 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549610 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:41.554250 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549615 5035 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549619 5035 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549624 5035 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549628 5035 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549632 5035 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549636 5035 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549640 5035 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549645 5035 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549649 5035 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549653 5035 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549658 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549662 5035 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549666 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549671 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549675 5035 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549680 5035 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549684 5035 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549688 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549693 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549698 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:41.554847 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549702 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549707 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549712 5035 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549716 5035 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549720 5035 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549724 5035 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549728 5035 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549732 5035 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549736 5035 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549741 5035 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549745 5035 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549749 5035 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549753 5035 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549760 5035 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549767 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549772 5035 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549777 5035 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549781 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549787 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549791 5035 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:41.555327 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549796 5035 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549800 5035 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549804 5035 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549809 5035 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549815 5035 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549821 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549826 5035 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549831 5035 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549837 5035 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549841 5035 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549846 5035 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549850 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549854 5035 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549858 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549863 5035 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549867 5035 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549871 5035 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549876 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549880 5035 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:41.555819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549885 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549890 5035 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549894 5035 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549898 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549902 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549907 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549911 5035 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549916 5035 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549920 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549925 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549929 5035 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549934 5035 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549938 5035 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549942 5035 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549947 5035 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549952 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549956 5035 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549960 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549964 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:41.556278 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549968 5035 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549972 5035 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549977 5035 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549981 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549985 5035 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549989 5035 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549993 5035 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.549998 5035 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550002 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550007 5035 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550012 5035 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550017 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550022 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550026 5035 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550030 5035 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550034 5035 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550038 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550048 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550052 5035 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550056 5035 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:41.556774 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550062 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.550066 5035 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550198 5035 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550210 5035 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550219 5035 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550225 5035 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550232 5035 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550238 5035 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550248 5035 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550255 5035 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550260 5035 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550265 5035 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550271 5035 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550276 5035 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550288 5035 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550293 5035 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550298 5035 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550302 5035 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550307 5035 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550312 5035 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550320 5035 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550325 5035 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550331 5035 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550336 5035 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:28:41.557243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550342 5035 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550348 5035 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550353 5035 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550358 5035 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550364 5035 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550369 5035 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550374 5035 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550379 5035 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550384 5035 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550388 5035 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550395 5035 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550400 5035 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550405 5035 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550410 5035 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550414 5035 flags.go:64] FLAG: --enable-server="true" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550419 5035 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550426 5035 flags.go:64] FLAG: --event-burst="100" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550431 5035 flags.go:64] FLAG: --event-qps="50" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550436 5035 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550440 5035 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550445 5035 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550452 5035 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550458 5035 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550464 5035 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550469 5035 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:28:41.557825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550474 5035 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550479 5035 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550483 5035 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550488 5035 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550492 5035 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550497 5035 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550503 5035 flags.go:64] FLAG: --feature-gates="" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550509 5035 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550514 5035 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550519 5035 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550540 5035 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550546 5035 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550552 5035 flags.go:64] FLAG: --help="false" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550556 5035 flags.go:64] FLAG: --hostname-override="ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550561 5035 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550566 5035 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550571 5035 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550577 5035 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550583 5035 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550588 5035 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550592 5035 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550597 5035 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550602 5035 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:28:41.558659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550606 5035 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550612 5035 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550616 5035 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550621 5035 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550626 5035 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550631 5035 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550638 5035 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550643 5035 flags.go:64] FLAG: --lock-file="" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550647 5035 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550652 5035 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550657 5035 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550666 5035 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550671 5035 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550676 5035 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550680 5035 flags.go:64] FLAG: --logging-format="text" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550686 5035 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550692 5035 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550697 5035 flags.go:64] FLAG: --manifest-url="" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550702 5035 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550710 5035 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550715 5035 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550721 5035 flags.go:64] FLAG: --max-pods="110" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550726 5035 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550731 5035 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550736 5035 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:28:41.559507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550741 5035 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550746 5035 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550751 5035 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550756 5035 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550766 5035 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550773 5035 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550778 5035 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550783 5035 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550788 5035 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550792 5035 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550798 5035 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550803 5035 flags.go:64] FLAG: --port="10250" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550808 5035 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550815 5035 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07cc7ab149845010e" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550820 5035 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550826 5035 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550831 5035 flags.go:64] FLAG: --register-node="true" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550835 5035 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550841 5035 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550846 5035 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550850 5035 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550855 5035 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550862 5035 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550867 5035 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550873 5035 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550877 5035 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:28:41.560220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550883 5035 flags.go:64] FLAG: --runonce="false" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550888 5035 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550893 5035 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550897 5035 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550902 5035 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550906 5035 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550911 5035 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550916 5035 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550920 5035 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550925 5035 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550929 5035 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550934 5035 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550939 5035 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550944 5035 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550949 5035 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550954 5035 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550962 5035 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550967 5035 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550971 5035 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550981 5035 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550985 5035 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550990 5035 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.550995 5035 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.551000 5035 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.551005 5035 flags.go:64] FLAG: --v="2" Oct 30 01:28:41.560865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.551011 5035 flags.go:64] FLAG: --version="false" Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.551018 5035 flags.go:64] FLAG: --vmodule="" Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.551025 5035 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.551030 5035 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551241 5035 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551248 5035 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551254 5035 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551261 5035 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551265 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551270 5035 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551274 5035 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551279 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551283 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551288 5035 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551292 5035 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551297 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551301 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551306 5035 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551310 5035 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551315 5035 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:41.561563 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551320 5035 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551324 5035 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551328 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551333 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551337 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551341 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551347 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551351 5035 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551356 5035 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551360 5035 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551364 5035 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551369 5035 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551373 5035 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551378 5035 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551382 5035 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551386 5035 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551390 5035 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551395 5035 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551399 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:41.562254 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551406 5035 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551410 5035 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551414 5035 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551418 5035 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551423 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551427 5035 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551434 5035 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551439 5035 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551444 5035 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551448 5035 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551452 5035 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551457 5035 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551461 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551466 5035 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551470 5035 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551474 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551479 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551483 5035 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551488 5035 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551495 5035 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:41.562842 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551499 5035 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551503 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551507 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551512 5035 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551516 5035 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551521 5035 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551542 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551547 5035 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551551 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551555 5035 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551563 5035 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551569 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551576 5035 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551583 5035 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551589 5035 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551594 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551599 5035 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551603 5035 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:41.563324 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551608 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551612 5035 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551617 5035 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551621 5035 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551625 5035 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551630 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551634 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551638 5035 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551643 5035 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551647 5035 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551652 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551656 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551661 5035 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551668 5035 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551672 5035 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551676 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551681 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551685 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551690 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551694 5035 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:41.563811 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551698 5035 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551703 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551707 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551712 5035 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551716 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551720 5035 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551727 5035 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551731 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551737 5035 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551741 5035 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551745 5035 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551749 5035 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551754 5035 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.551796 5035 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551858 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551866 5035 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:41.564436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551871 5035 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551875 5035 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551880 5035 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551884 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551889 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551893 5035 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551897 5035 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551902 5035 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551906 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551912 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551917 5035 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551921 5035 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551926 5035 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551930 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551935 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551939 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551944 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551948 5035 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551952 5035 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551957 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:41.564918 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551961 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551965 5035 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551969 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551975 5035 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551980 5035 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551985 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551990 5035 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551995 5035 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.551999 5035 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552004 5035 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552008 5035 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552012 5035 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552017 5035 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552021 5035 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552025 5035 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552030 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552034 5035 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552039 5035 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552043 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552048 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:41.565394 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552052 5035 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552058 5035 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552063 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552067 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552072 5035 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552076 5035 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552081 5035 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552085 5035 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552090 5035 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552094 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552098 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552103 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552107 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552112 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552116 5035 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552122 5035 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552126 5035 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552131 5035 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552136 5035 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552140 5035 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:41.565904 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552144 5035 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552149 5035 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552153 5035 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552158 5035 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552162 5035 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552166 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552170 5035 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552175 5035 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552179 5035 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552183 5035 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552187 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552192 5035 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552196 5035 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552202 5035 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552207 5035 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552211 5035 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552216 5035 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552220 5035 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552224 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552229 5035 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:41.566412 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552233 5035 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552237 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552242 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552248 5035 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552252 5035 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552256 5035 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552261 5035 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552267 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552271 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552276 5035 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552280 5035 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552286 5035 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552290 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552295 5035 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552299 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552303 5035 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552310 5035 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552317 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552322 5035 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:41.566934 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552329 5035 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552336 5035 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552341 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552345 5035 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.552350 5035 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.555987 5035 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.556003 5035 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.556022 5035 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.556027 5035 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556082 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556087 5035 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556091 5035 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556094 5035 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556099 5035 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556103 5035 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556107 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556109 5035 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556112 5035 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:41.567403 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556115 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556118 5035 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556121 5035 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556123 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556126 5035 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556129 5035 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556131 5035 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556134 5035 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556137 5035 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556139 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556142 5035 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556144 5035 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556147 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556150 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556152 5035 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556155 5035 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556158 5035 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556162 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556165 5035 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556167 5035 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:41.568047 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556170 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556173 5035 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556176 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556178 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556181 5035 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556183 5035 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556186 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556189 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556191 5035 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556194 5035 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556196 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556199 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556201 5035 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556204 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556206 5035 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556209 5035 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556212 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556215 5035 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556217 5035 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556220 5035 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:41.568553 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556222 5035 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556225 5035 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556227 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556230 5035 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556233 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556235 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556238 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556241 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556244 5035 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556247 5035 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556250 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556253 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556255 5035 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556258 5035 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556260 5035 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556263 5035 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556266 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556268 5035 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556271 5035 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:41.569022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556273 5035 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556276 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556278 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556281 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556284 5035 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556286 5035 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556289 5035 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556292 5035 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556294 5035 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556297 5035 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556299 5035 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556302 5035 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556304 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556307 5035 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556309 5035 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556312 5035 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556314 5035 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556317 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556319 5035 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:41.569476 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556322 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556325 5035 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556328 5035 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556331 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556334 5035 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556337 5035 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556341 5035 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556344 5035 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556347 5035 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556350 5035 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556353 5035 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556356 5035 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556359 5035 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556361 5035 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556364 5035 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556366 5035 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556369 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556372 5035 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:41.569940 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556374 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.556401 5035 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556546 5035 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556552 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556555 5035 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556557 5035 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556560 5035 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556563 5035 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556566 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556569 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556571 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556574 5035 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556577 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556579 5035 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556582 5035 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556585 5035 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:41.570365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556588 5035 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556590 5035 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556593 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556596 5035 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556599 5035 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556602 5035 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556604 5035 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556607 5035 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556609 5035 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556612 5035 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556614 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556617 5035 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556620 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556623 5035 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556625 5035 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556628 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556630 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556633 5035 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556635 5035 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556638 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:41.570839 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556640 5035 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556643 5035 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556645 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556647 5035 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556650 5035 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556652 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556655 5035 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556657 5035 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556660 5035 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556663 5035 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556665 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556668 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556671 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556673 5035 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556676 5035 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556679 5035 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556681 5035 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556684 5035 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556687 5035 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556690 5035 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:41.571401 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556692 5035 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556695 5035 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556697 5035 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556700 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556703 5035 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556705 5035 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556708 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556710 5035 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556713 5035 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556715 5035 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556718 5035 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556724 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556727 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556730 5035 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556732 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556736 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556738 5035 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556741 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556744 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:41.571911 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556748 5035 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556752 5035 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556755 5035 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556758 5035 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556761 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556764 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556766 5035 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556769 5035 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556772 5035 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556775 5035 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556777 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556781 5035 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556784 5035 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556787 5035 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556790 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556793 5035 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556797 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556800 5035 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556802 5035 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:41.572402 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556805 5035 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556807 5035 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556810 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556812 5035 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556815 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556818 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556821 5035 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556824 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556826 5035 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556829 5035 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556831 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556834 5035 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556836 5035 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556839 5035 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.556861 5035 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556893 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:41.572877 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556896 5035 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556900 5035 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556903 5035 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556905 5035 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556908 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556911 5035 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556913 5035 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556916 5035 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556918 5035 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556921 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556924 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556927 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556929 5035 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556932 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556934 5035 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556937 5035 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556939 5035 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556942 5035 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556945 5035 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556948 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:41.573259 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556951 5035 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556953 5035 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556956 5035 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556958 5035 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556961 5035 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556963 5035 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556966 5035 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556968 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556970 5035 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556973 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556976 5035 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556978 5035 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556981 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556984 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556986 5035 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556989 5035 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556991 5035 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556994 5035 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556996 5035 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.556999 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:41.573805 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557001 5035 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557004 5035 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557007 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557009 5035 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557012 5035 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557015 5035 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557017 5035 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557020 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557022 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557025 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557028 5035 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557030 5035 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557033 5035 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557035 5035 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557038 5035 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557041 5035 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557043 5035 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557046 5035 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557048 5035 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557050 5035 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:41.574290 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557053 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557056 5035 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557058 5035 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557061 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557063 5035 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557066 5035 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557069 5035 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557071 5035 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557074 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557076 5035 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557079 5035 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557081 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557084 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557086 5035 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557089 5035 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557092 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557096 5035 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557100 5035 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557103 5035 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:41.574820 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557106 5035 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557108 5035 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557111 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557114 5035 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557116 5035 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557119 5035 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557121 5035 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557124 5035 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557127 5035 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557131 5035 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557134 5035 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557136 5035 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557141 5035 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557144 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557148 5035 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557151 5035 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557154 5035 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557157 5035 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557159 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:41.575404 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557163 5035 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557165 5035 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557168 5035 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557170 5035 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557173 5035 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557175 5035 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:41.557178 5035 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.557224 5035 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.558512 5035 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.558611 5035 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.558929 5035 server.go:1014] "Starting client certificate rotation" Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.559282 5035 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.559505 5035 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:23:01 +0000 UTC" deadline="2028-04-10 21:18:52.804571781 +0000 UTC" Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.559582 5035 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12691h50m11.244994251s" Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.559887 5035 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.560889 5035 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:28:41.576201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.562039 5035 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:28:41.576891 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.563661 5035 log.go:25] "Validated CRI v1 image API" Oct 30 01:28:41.576891 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.564370 5035 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:28:41.576891 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.566262 5035 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ec2340c-3279-43d2-83d0-34fd083e8244:/dev/nvme0n1p4 aa7532c3-5652-44b6-9b48-e764a5cf4ac9:/dev/nvme0n1p3] Oct 30 01:28:41.577046 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.566288 5035 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0caff05880f97cccb5aba759883387b750539454048491c4445102cb3efdcf9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0caff05880f97cccb5aba759883387b750539454048491c4445102cb3efdcf9a/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26bd9805c1799a342d75b1d2cb26c7355e0562bbc24b77ffc6da74112c481692/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26bd9805c1799a342d75b1d2cb26c7355e0562bbc24b77ffc6da74112c481692/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34b6335dff88b63b558f7923f9ba4b65fec5e0e761f669a0786ebdaf281e4896/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34b6335dff88b63b558f7923f9ba4b65fec5e0e761f669a0786ebdaf281e4896/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39532e095dc6835edc2016e5c2ac9dae41e2c3b7b23f1ec08983b1231a326ce0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39532e095dc6835edc2016e5c2ac9dae41e2c3b7b23f1ec08983b1231a326ce0/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bd27ddb4aadd0f011f371665ead72daa7d96d13bbb970897f7ba7f8e7cc7a61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bd27ddb4aadd0f011f371665ead72daa7d96d13bbb970897f7ba7f8e7cc7a61/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d48388582a3edbde0b946e0c03f5e00d19b80127b216772a8a11eef22beba85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d48388582a3edbde0b946e0c03f5e00d19b80127b216772a8a11eef22beba85/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/504c278b298d00620d18e633174d51c060a7cb17ff206253943c4dbd1d57a5e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/504c278b298d00620d18e633174d51c060a7cb17ff206253943c4dbd1d57a5e4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54475ade28d87774d9457a4878e1f2f8aa10364657f4ec57c9c6a2e3ffbee909/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54475ade28d87774d9457a4878e1f2f8aa10364657f4ec57c9c6a2e3ffbee909/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/553d0043bae899bc3e7d069be69f093ec235ea2d9ec9086ab8ac2b09ebcb2feb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/553d0043bae899bc3e7d069be69f093ec235ea2d9ec9086ab8ac2b09ebcb2feb/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58efa8b6b6b953c92c4289f1160ba63e37553f2d48af5a0f206aed7a1971ac2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58efa8b6b6b953c92c4289f1160ba63e37553f2d48af5a0f206aed7a1971ac2e/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66a81e642a8bec8062afdc1fe7c39e670910e843991ee1c072443e19ed47e2a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66a81e642a8bec8062afdc1fe7c39e670910e843991ee1c072443e19ed47e2a8/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c3e87c3c22046833f4b5eeafcdf36c1842b846590bc4a8f0fbf0dfa300d30c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c3e87c3c22046833f4b5eeafcdf36c1842b846590bc4a8f0fbf0dfa300d30c5/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84c6e07766b7ee19cabc5209685470de32cd0e0f92448ba7eaf3d31b6411bd28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84c6e07766b7ee19cabc5209685470de32cd0e0f92448ba7eaf3d31b6411bd28/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c20636b674ce676c941dae5a18613e2704ef4a4fa6e46ed8f356649665f9da3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c20636b674ce676c941dae5a18613e2704ef4a4fa6e46ed8f356649665f9da3/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a050cd3b4e5df5ce962f0676b2f2386ef0e8184b58a3943f81bc084779c0126d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a050cd3b4e5df5ce962f0676b2f2386ef0e8184b58a3943f81bc084779c0126d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1e24973d5ca651624d9c2eebbbacceb04eb96da53bb6658edeed8426dcba646/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1e24973d5ca651624d9c2eebbbacceb04eb96da53bb6658edeed8426dcba646/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9ea3f0b439c9e28fa22a2943975ef5c6e1f577c3598a1a690b561831c0952af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9ea3f0b439c9e28fa22a2943975ef5c6e1f577c3598a1a690b561831c0952af/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9efa8322afd0ea236d056ab5cffd195e3d438424e609da773477fccfdce7c05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9efa8322afd0ea236d056ab5cffd195e3d438424e609da773477fccfdce7c05/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0db4484c8069d40507da09a5544e6a44216118be03db395b8a3232f20c452af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0db4484c8069d40507da09a5544e6a44216118be03db395b8a3232f20c452af/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd28e5fba1f493a86f011eda2f7e7bd0033d46c91907e1870e063ea700a59417/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd28e5fba1f493a86f011eda2f7e7bd0033d46c91907e1870e063ea700a59417/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d12efdef0c32486f8df86324b241438773d6318337ca920f2cc7b474630fe4b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d12efdef0c32486f8df86324b241438773d6318337ca920f2cc7b474630fe4b5/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df1689303ba6d4314fe08e797680b367740df33fd1d7da9f114eb0ff6312e878/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df1689303ba6d4314fe08e797680b367740df33fd1d7da9f114eb0ff6312e878/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e016e044e47012216c57c1498de711374dbfac8918dfbdb2e268162c3f9b5943/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e016e044e47012216c57c1498de711374dbfac8918dfbdb2e268162c3f9b5943/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e628113131fd5a44909ada19ea8ae173a532d988aacb998698aff45a9fc8e424/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e628113131fd5a44909ada19ea8ae173a532d988aacb998698aff45a9fc8e424/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e93c31328aadee075019623b5e993e1754e19f391920cbf8d666f671b7ef129b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e93c31328aadee075019623b5e993e1754e19f391920cbf8d666f671b7ef129b/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e964e4d9a4ea604160731a35bc55ad8b8d907ce3c62b8b19c1ad2268bf355772/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e964e4d9a4ea604160731a35bc55ad8b8d907ce3c62b8b19c1ad2268bf355772/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ab5bdf-56b9-4176-af0e-f41d92760b84/volumes/kubernetes.io~projected/kube-api-access-f72c7:{mountpoint:/var/lib/kubelet/pods/02ab5bdf-56b9-4176-af0e-f41d92760b84/volumes/kubernetes.io~projected/kube-api-access-f72c7 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ab5bdf-56b9-4176-af0e-f41d92760b84/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/02ab5bdf-56b9-4176-af0e-f41d92760b84/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ab5bdf-56b9-4176-af0e-f41d92760b84/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/02ab5bdf-56b9-4176-af0e-f41d92760b84/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08cfb9ab-35cb-4695-8758-71f71f0c34c3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/08cfb9ab-35cb-4695-8758-71f71f0c34c3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08cfb9ab-35cb-4695-8758-71f71f0c34c3/volumes/kubernetes.io~projected/kube-api-access-tjdvd:{mountpoint:/var/lib/kubelet/pods/08cfb9ab-35cb-4695-8758-71f71f0c34c3/volumes/kubernetes.io~projected/kube-api-access-tjdvd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08cfb9ab-35cb-4695-8758-71f71f0c34c3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/08cfb9ab-35cb-4695-8758-71f71f0c34c3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10d07cdc-86b5-4181-9b95-9cfb0341db09/volumes/kubernetes.io~projected/kube-api-access-4hrqt:{mountpoint:/var/lib/kubelet/pods/10d07cdc-86b5-4181-9b95-9cfb0341db09/volumes/kubernetes.io~projected/kube-api-access-4hrqt major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/118aebf2-95cc-4c0a-8f6d-fed0c906316d/volumes/kubernetes.io~projected/kube-api-access-9c5xd:{mountpoint:/var/lib/kubelet/pods/118aebf2-95cc-4c0a-8f6d-fed0c906316d/volumes/kubernetes.io~projected/kube-api-access-9c5xd major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1989a7c0-f591-413a-b111-266ad115c63a/volumes/kubernetes.io~projected/kube-api-access-8hmfr:{mountpoint:/var/lib/kubelet/pods/1989a7c0-f591-413a-b111-266ad115c63a/volumes/kubernetes.io~projected/kube-api-access-8hmfr major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2160cefb-5ed3-4ee9-abc4-9184f7825317/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2160cefb-5ed3-4ee9-abc4-9184f7825317/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2160cefb-5ed3-4ee9-abc4-9184f7825317/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2160cefb-5ed3-4ee9-abc4-9184f7825317/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2160cefb-5ed3-4ee9-abc4-9184f7825317/volumes/kubernetes.io~projected/kube-api-access-q2qx7:{mountpoint:/var/lib/kubelet/pods/2160cefb-5ed3-4ee9-abc4-9184f7825317/volumes/kubernetes.io~projected/kube-api-access-q2qx7 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22ff8c4e-b07d-417b-b4d3-61560b014b77/volumes/kubernetes.io~projected/kube-api-access-ddck4:{mountpoint:/var/lib/kubelet/pods/22ff8c4e-b07d-417b-b4d3-61560b014b77/volumes/kubernetes.io~projected/kube-api-access-ddck4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2986b483-9c1e-4d59-89d5-01e287807f8a/volumes/kubernetes.io~projected/kube-api-access-6qsv9:{mountpoint:/var/lib/kubelet/pods/2986b483-9c1e-4d59-89d5-01e287807f8a/volumes/kubernetes.io~projected/kube-api-access-6qsv9 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32c651f8-7083-4335-a937-682d7a665312/volumes/kubernetes.io~projected/kube-api-access-97hrx:{mountpoint:/var/lib/kubelet/pods/32c651f8-7083-4335-a937-682d7a665312/volumes/kubernetes.io~projected/kube-api-access-97hrx major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34fec9a2-a544-4181-bf3a-b3e81ea7929f/volumes/kubernetes.io~projected/kube-api-access-6gsmz:{mountpoint:/var/lib/kubelet/pods/34fec9a2-a544-4181-bf3a-b3e81ea7929f/volumes/kubernetes.io~projected/kube-api-access-6gsmz major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34fec9a2-a544-4181-bf3a-b3e81ea7929f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/34fec9a2-a544-4181-bf3a-b3e81ea7929f/volumes/kubernetes.io~secret/cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c903b8f-507e-4dec-801e-3213298d7e0a/volumes/kubernetes.io~projected/kube-api-access-7zsmr:{mountpoint:/var/lib/kubelet/pods/3c903b8f-507e-4dec-801e-3213298d7e0a/volumes/kubernetes.io~projected/kube-api-access-7zsmr major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c903b8f-507e-4dec-801e-3213298d7e0a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3c903b8f-507e-4dec-801e-3213298d7e0a/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e3639d2-bb67-4554-b567-be193af33780/volumes/kubernetes.io~projected/kube-api-access-wvrtd:{mountpoint:/var/lib/kubelet/pods/3e3639d2-bb67-4554-b567-be193af33780/volumes/kubernetes.io~projected/kube-api-access-wvrtd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44ef0b6d-c29b-4925-b06f-5136ef9aa906/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/44ef0b6d-c29b-4925-b06f-5136ef9aa906/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53172623-6cce-4589-91c3-811476ab540d/volumes/kubernetes.io~projected/kube-api-access-d9xrd:{mountpoint:/var/lib/kubelet/pods/53172623-6cce-4589-91c3-811476ab540d/volumes/kubernetes.io~projected/kube-api-access-d9xrd major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53abc737-fbef-4574-96b8-b9adee4fade2/volumes/kubernetes.io~projected/kube-api-access-7ntm6:{mountpoint:/var/lib/kubelet/pods/53abc737-fbef-4574-96b8-b9adee4fade2/volumes/kubernetes.io~projected/kube-api-access-7ntm6 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cafa88a-427e-45c7-a706-d56fd5425610/volumes/kubernetes.io~projected/kube-api-access-bd9vt:{mountpoint:/var/lib/kubelet/pods/5cafa88a-427e-45c7-a706-d56fd5425610/volumes/kubernetes.io~projected/kube-api-access-bd9vt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a968163-1344-4372-a8b9-bf896d90e6a9/volumes/kubernetes.io~projected/kube-api-access-x9xkz:{mountpoint:/var/lib/kubelet/pods/6a968163-1344-4372-a8b9-bf896d90e6a9/volumes/kubernetes.io~projected/kube-api-access-x9xkz major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a968163-1344-4372-a8b9-bf896d90e6a9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6a968163-1344-4372-a8b9-bf896d90e6a9/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e60155d-a79a-453f-a17a-902840cdeb14/volumes/kubernetes.io~projected/kube-api-access-c7p77:{mountpoint:/var/lib/kubelet/pods/6e60155d-a79a-453f-a17a-902840cdeb14/volumes/kubernetes.io~projected/kube-api-access-c7p77 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~projected/kube-api-access-qqx8q:{mountpoint:/var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~projected/kube-api-access-qqx8q major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c481c3b-feae-4510-83c5-0dc08ce51db0/volumes/kubernetes.io~projected/kube-api-access-mrh86:{mountpoint:/var/lib/kubelet/pods/7c481c3b-feae-4510-83c5-0dc08ce51db0/volumes/kubernetes.io~projected/kube-api-access-mrh86 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c481c3b-feae-4510-83c5-0dc08ce51db0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7c481c3b-feae-4510-83c5-0dc08ce51db0/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~projected/kube-api-access-9fgwl:{mountpoint:/var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~projected/kube-api-access-9fgwl major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/971584c9-502d-4dfa-beae-8d8d7ae88cb3/volumes/kubernetes.io~projected/kube-api-access-2bwqn:{mountpoint:/var/lib/kubelet/pods/971584c9-502d-4dfa-beae-8d8d7ae88cb3/volumes/kubernetes.io~projected/kube-api-access-2bwqn major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8ac7159-dcc1-46dc-bae0-eae434811dbe/volumes/kubernetes.io~projected/kube-api-access-gp748:{mountpoint:/var/lib/kubelet/pods/a8ac7159-dcc1-46dc-bae0-eae434811dbe/volumes/kubernetes.io~projected/kube-api-access-gp748 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8ac7159-dcc1-46dc-bae0-eae434811dbe/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a8ac7159-dcc1-46dc-bae0-eae434811dbe/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aad411a5-bb79-4e0c-abd3-85c5ba535c93/volumes/kubernetes.io~projected/kube-api-access-bx95x:{mountpoint:/var/lib/kubelet/pods/aad411a5-bb79-4e0c-abd3-85c5ba535c93/volumes/kubernetes.io~projected/kube-api-access-bx95x major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf8cbe6-582e-408b-85a9-d969e18b4682/volumes/kubernetes.io~projected/kube-api-access-4ql9j:{mountpoint:/var/lib/kubelet/pods/acf8cbe6-582e-408b-85a9-d969e18b4682/volumes/kubernetes.io~projected/kube-api-access-4ql9j major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b011ba7c-f34c-44e2-9498-5bd998c41b07/volumes/kubernetes.io~projected/kube-api-access-9bdfx:{mountpoint:/var/lib/kubelet/pods/b011ba7c-f34c-44e2-9498-5bd998c41b07/volumes/kubernetes.io~projected/kube-api-access-9bdfx major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b011ba7c-f34c-44e2-9498-5bd998c41b07/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/b011ba7c-f34c-44e2-9498-5bd998c41b07/volumes/kubernetes.io~secret/signing-key major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b82d0410-941f-43e4-9ae9-5d37f9c409be/volumes/kubernetes.io~projected/kube-api-access-79xfg:{mountpoint:/var/lib/kubelet/pods/b82d0410-941f-43e4-9ae9-5d37f9c409be/volumes/kubernetes.io~projected/kube-api-access-79xfg major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd24e16c-c31b-4af0-b518-cfe8c85937ac/volumes/kubernetes.io~projected/kube-api-access-t8m9r:{mountpoint:/var/lib/kubelet/pods/bd24e16c-c31b-4af0-b518-cfe8c85937ac/volumes/kubernetes.io~projected/kube-api-access-t8m9r major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c36ce70e-acde-4698-89b9-37d85639a669/volumes/kubernetes.io~projected/kube-api-access-78x8b:{mountpoint:/var/lib/kubelet/pods/c36ce70e-acde-4698-89b9-37d85639a669/volumes/kubernetes.io~projected/kube-api-access-78x8b major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca1fbc85-1496-4e79-9bc9-330a8540a39e/volumes/kubernetes.io~projected/kube-api-access-rtlvp:{mountpoint:/var/lib/kubelet/pods/ca1fbc85-1496-4e79-9bc9-330a8540a39e/volumes/kubernetes.io~projected/kube-api-access-rtlvp major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eadbf8a4-af76-4fab-b804-13e1ac18afcc/volumes/kubernetes.io~projected/kube-api-access-f2hb2:{mountpoint:/var/lib/kubelet/pods/eadbf8a4-af76-4fab-b804-13e1ac18afcc/volumes/kubernetes.io~projected/kube-api-access-f2hb2 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdd98c23-760f-487d-b8d2-be0684665b55/volumes/kubernetes.io~projected/kube-api-access-8jm4t:{mountpoint:/var/lib/kubelet/pods/fdd98c23-760f-487d-b8d2-be0684665b55/volumes/kubernetes.io~projected/kube-api-access-8jm4t major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/69e200b9433bfeb300560b32353fa651de8c8ba5f5c461ded0453ebce0c81257/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0bd1085e6a8c8e98264b9ea35b2bdceac729f0ad8cddb1856248d191f4525a92/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f759b27056fca7cca524867d03bb67e027486bd2f33f4b2bbdcbff64b6b67b1e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/effe0ec38af0bf52e05de66dcab2a693d9197fff56022593bff02b952c77a3ec/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6bfe5ad35fbb71da65bc3d54658fca10d872c3324287c970bae546e512be4a38/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/971354e5ec95cdefe32efcbd7fea4b15d25bbafb8cd7b964b7983e2ec6b0d916/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4a5481aa10d7053b78ac91edb1478683938f5b502956e398cd1a070526652f05/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c77710de29455317354a8d7285b93eecda8d13ae1b6da5d5412de4c19d9ede90/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3e9bde42113c2c382a294b95163d56afbfb400b668bfb7bdf5c40bfbadb0934a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/42f6650ab6f9605e3c3a162c125e068f8474a8f757e957b9bbca62dd27bf3ea8/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/427cf64d23b9c028b31cafef7b277120d89b4b91a23fe2517efb0b61bc8a03af/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/ae6f75a32bbf02652f8cd59a99c42f0c69dfa9eceb7f194b9738ba9fe285d53a/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9630a8bac8152a78b4061cb0c122fa7deb50440b46632e842c8eba968516fcb6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2c27fae4c2acd666001643c51f97c7cdd9c82ff0f75b019a8033576c8e8c8bdb/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1a29a0dc9efa5c2e6a1dfd8af89d85ec9b83ce483cd3936630d85a8a362556e7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c7310f44a7b7fdbf19306e95bc737414849aac7d651af4bf2bffc11a3b7af999/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a394c0b3aad7a78b3a1b74653f32109ef4407bd34e177b49c7ac9e98b0e14170/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3902f7818742d182d4547df514eca38d6862b8149d2759701691ab0dc5d22f75/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/52f2517b9cca45963e2c8eb5193dbbad696a2900583f676dfd1c7471b8ec5704/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/38aeba8a889cbde1098fc1fe2888d4d26b4a6f506527a49e8479b35ca9f0a5a6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/080c4f41150a415bc52ff731bc9c22ec196dd4f51ad598582eba4ac61e6b056e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/1e59249c46e29d2de8fdb1af6520daddbd9d0b8625554e59f26f793cca4ae0d9/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/861103af30af840ba4d051c4e7930f2ea208e02690d2f2091a37396559baf7c0/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/6e830f08ab75828e7a102dffae28d879df4aefab2a62eaa533bdfeec7a366943/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/89996e4a33b3c02cbbe4b2e0aa9304a34e25f5b2670cad4bb44731666e08b873/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f2094697c3a9c83905d3927d48c62718a341078f5803c5d9b5b86be6b5ff393c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f386b13faceca248049e2eb1aca220b9b26377bc06e7d4ec624faa1d9894cafc/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/22d3706e48821dddb984132db3286dfbbd8031a79bc91e050818d6be3b640e16/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/80677c0de5467b4794ae94e9115de44a481d88a7848436cba83037f9fe7555ed/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/2888fa0d58478bf68d86216df47566a951b81f27077108e42e2c07eebeb55b26/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/29d7c735e4b727d7ccf587a5599fdf33228eb48b6840c95abf2665cafad4a6d5/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/89d1c0a2b24b278c58045e1bdf1c75c21ef0ad0d3e2fd5e0ebdb39099a3013e7/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/93608150bb6025a3f8e268979c2a0628a55998901eda8d4cffb652b711b19415/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/312114a8dc138797772f8f67dea4b8333f4d9561886e87089d93054e9f76f834/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/d138b6132d36192ea30f2c3c51e9fcc6dbfebc3b4f666192fdff8e46039cb9c7/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/45938f0545842759180e2bca73a0cd4487f4caa8b513629fa75c15aae66ea98a/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/cd6fb161569f5d01098931bb18caa8655b6d914d121f9db0e4f24dcbd9a0f05d/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/041139391ea76ba82776ecdacb4802fb8dbb2332b066d7b4a0b1270fb4db5397/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/a7eb7aba732225f6c2c4f87ef2d3275a51f71ac7c2410f7438f762731945359c/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/4edad6956b8963c462bfab9f41b239fd1bbab8e68dcbb4d6e7572c2f0a061719/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/7d74438b0dbc5d6805bbd325088399d5c3765b13391f638fe45be5a91e459b0d/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/9e33ee607398d1cb078883a70c2962f0ae78c66880e7431dbef3ff9dc3e52d01/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/27db164540b2800823d7fe1664e02e7423b6dd9c0c547bc699763f5761a626c7/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/4e73e25b1bcd640457a98ef46a9fa9c54212995d6ad65cafb8f73a729c7c33c9/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/935fad24627359aa70bcb51d674a7c34232b38700911fbfef28e8304769c871d/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/c593b0ab001e180c679fe925704c47de9546dd42e5adfa64c9347838e0bd5f1c/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/2f82a4427bbf5e85efa128a9d16a46ff4b93a71732e65e9377693fb08fdbbf4b/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/936380f575aaa4807d6702245afb08344ff1e9ac9c449c328c61a74ad576280d/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/7f92137b75cc439ec5a37de8e13f5a90019b5900e6ecd52959bb3772c27e1292/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/5c2dcad2f5785d743acc37b31e58042dcc60cd9e6ae91ae13194fea9b46b98f6/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/e41849efb7d360829fa6c19ab608579a780760f2acb44755ece7d34f18129155/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/6843e499099177ec31da51a2b43ff29b6534103bdac4a11def82962869c5f128/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/a023f39ae473368d1d5e2cf4a657e709b79dd29e85e900f706f8496a5e0e3a07/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/21d984f03c6d7dc17cd7962335016f22d514047d4063d40439ccbc7ade89b9c1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ff13585b6657bf2210177d27eb7255d467bca38ce0d6d2a8a6314d9a1eb8d454/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/930d99594bfcd84c90fc83c0f3d5a7f0842a3b282243cadca25acf129afc227d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/d562dfcdc89e17adeb23ed1fbfca4d7a8d3623c91e468cd9b3ceb01e7e18d151/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/edf2b56489f9ce01669520057bf850c8a111c60b39f004f38730844770748c1b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/632e431a03f83fef1a018e6d86098b23e8cf8c7fe9993d4fbd6241bd2530c943/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/5afba897f76fc770c749f704b7eaaeeb52a934d4186e97ba5f3f2cfa03c2bfe3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c12e886fa39282e762970ecb03ff9f7afb15d394e3e8c3d6b8734ae70bc67663/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a3a4401f6cd43e9a3a2a01f3960a8de809fad2a3d1c807185f6510caf1432a42/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.574674 5035 manager.go:217] Machine: {Timestamp:2026-10-30 01:28:41.574198783 +0000 UTC m=+0.077130994 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099389 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec247cd7daafc93fef2201935a2b7ea3 SystemUUID:ec247cd7-daaf-c93f-ef22-01935a2b7ea3 BootID:e71f8d96-feab-4af0-847e-6e65d4db1120 Filesystems:[{Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0caff05880f97cccb5aba759883387b750539454048491c4445102cb3efdcf9a/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66a81e642a8bec8062afdc1fe7c39e670910e843991ee1c072443e19ed47e2a8/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02ab5bdf-56b9-4176-af0e-f41d92760b84/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3c903b8f-507e-4dec-801e-3213298d7e0a/volumes/kubernetes.io~projected/kube-api-access-7zsmr DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5cafa88a-427e-45c7-a706-d56fd5425610/volumes/kubernetes.io~projected/kube-api-access-bd9vt DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c20636b674ce676c941dae5a18613e2704ef4a4fa6e46ed8f356649665f9da3/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02ab5bdf-56b9-4176-af0e-f41d92760b84/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39532e095dc6835edc2016e5c2ac9dae41e2c3b7b23f1ec08983b1231a326ce0/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d48388582a3edbde0b946e0c03f5e00d19b80127b216772a8a11eef22beba85/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53abc737-fbef-4574-96b8-b9adee4fade2/volumes/kubernetes.io~projected/kube-api-access-7ntm6 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd24e16c-c31b-4af0-b518-cfe8c85937ac/volumes/kubernetes.io~projected/kube-api-access-t8m9r DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b011ba7c-f34c-44e2-9498-5bd998c41b07/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:350 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a968163-1344-4372-a8b9-bf896d90e6a9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b82d0410-941f-43e4-9ae9-5d37f9c409be/volumes/kubernetes.io~projected/kube-api-access-79xfg DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c481c3b-feae-4510-83c5-0dc08ce51db0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/553d0043bae899bc3e7d069be69f093ec235ea2d9ec9086ab8ac2b09ebcb2feb/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e016e044e47012216c57c1498de711374dbfac8918dfbdb2e268162c3f9b5943/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e964e4d9a4ea604160731a35bc55ad8b8d907ce3c62b8b19c1ad2268bf355772/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/acf8cbe6-582e-408b-85a9-d969e18b4682/volumes/kubernetes.io~projected/kube-api-access-4ql9j DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~projected/kube-api-access-qqx8q DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/118aebf2-95cc-4c0a-8f6d-fed0c906316d/volumes/kubernetes.io~projected/kube-api-access-9c5xd DeviceMajor:0 DeviceMinor:321 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08cfb9ab-35cb-4695-8758-71f71f0c34c3/volumes/kubernetes.io~projected/kube-api-access-tjdvd DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34fec9a2-a544-4181-bf3a-b3e81ea7929f/volumes/kubernetes.io~projected/kube-api-access-6gsmz DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df1689303ba6d4314fe08e797680b367740df33fd1d7da9f114eb0ff6312e878/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bd27ddb4aadd0f011f371665ead72daa7d96d13bbb970897f7ba7f8e7cc7a61/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a050cd3b4e5df5ce962f0676b2f2386ef0e8184b58a3943f81bc084779c0126d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/53172623-6cce-4589-91c3-811476ab540d/volumes/kubernetes.io~projected/kube-api-access-d9xrd DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e628113131fd5a44909ada19ea8ae173a532d988aacb998698aff45a9fc8e424/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1e24973d5ca651624d9c2eebbbacceb04eb96da53bb6658edeed8426dcba646/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aad411a5-bb79-4e0c-abd3-85c5ba535c93/volumes/kubernetes.io~projected/kube-api-access-bx95x DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8ac7159-dcc1-46dc-bae0-eae434811dbe/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/971584c9-502d-4dfa-beae-8d8d7ae88cb3/volumes/kubernetes.io~projected/kube-api-access-2bwqn DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08cfb9ab-35cb-4695-8758-71f71f0c34c3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a8ac7159-dcc1-46dc-bae0-eae434811dbe/volumes/kubernetes.io~projected/kube-api-access-gp748 DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02ab5bdf-56b9-4176-af0e-f41d92760b84/volumes/kubernetes.io~projected/kube-api-access-f72c7 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/2160cefb-5ed3-4ee9-abc4-9184f7825317/volumes/kubernetes.io~projected/kube-api-access-q2qx7 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e60155d-a79a-453f-a17a-902840cdeb14/volumes/kubernetes.io~projected/kube-api-access-c7p77 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54475ade28d87774d9457a4878e1f2f8aa10364657f4ec57c9c6a2e3ffbee909/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08cfb9ab-35cb-4695-8758-71f71f0c34c3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2986b483-9c1e-4d59-89d5-01e287807f8a/volumes/kubernetes.io~projected/kube-api-access-6qsv9 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1989a7c0-f591-413a-b111-266ad115c63a/volumes/kubernetes.io~projected/kube-api-access-8hmfr DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eadbf8a4-af76-4fab-b804-13e1ac18afcc/volumes/kubernetes.io~projected/kube-api-access-f2hb2 DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e93c31328aadee075019623b5e993e1754e19f391920cbf8d666f671b7ef129b/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84c6e07766b7ee19cabc5209685470de32cd0e0f92448ba7eaf3d31b6411bd28/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a968163-1344-4372-a8b9-bf896d90e6a9/volumes/kubernetes.io~projected/kube-api-access-x9xkz DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e3639d2-bb67-4554-b567-be193af33780/volumes/kubernetes.io~projected/kube-api-access-wvrtd DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2160cefb-5ed3-4ee9-abc4-9184f7825317/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd28e5fba1f493a86f011eda2f7e7bd0033d46c91907e1870e063ea700a59417/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44ef0b6d-c29b-4925-b06f-5136ef9aa906/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9efa8322afd0ea236d056ab5cffd195e3d438424e609da773477fccfdce7c05/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32c651f8-7083-4335-a937-682d7a665312/volumes/kubernetes.io~projected/kube-api-access-97hrx DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22ff8c4e-b07d-417b-b4d3-61560b014b77/volumes/kubernetes.io~projected/kube-api-access-ddck4 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca1fbc85-1496-4e79-9bc9-330a8540a39e/volumes/kubernetes.io~projected/kube-api-access-rtlvp DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9ea3f0b439c9e28fa22a2943975ef5c6e1f577c3598a1a690b561831c0952af/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c36ce70e-acde-4698-89b9-37d85639a669/volumes/kubernetes.io~projected/kube-api-access-78x8b DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c3e87c3c22046833f4b5eeafcdf36c1842b846590bc4a8f0fbf0dfa300d30c5/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10d07cdc-86b5-4181-9b95-9cfb0341db09/volumes/kubernetes.io~projected/kube-api-access-4hrqt DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26bd9805c1799a342d75b1d2cb26c7355e0562bbc24b77ffc6da74112c481692/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c903b8f-507e-4dec-801e-3213298d7e0a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0db4484c8069d40507da09a5544e6a44216118be03db395b8a3232f20c452af/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34b6335dff88b63b558f7923f9ba4b65fec5e0e761f669a0786ebdaf281e4896/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fdd98c23-760f-487d-b8d2-be0684665b55/volumes/kubernetes.io~projected/kube-api-access-8jm4t DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2160cefb-5ed3-4ee9-abc4-9184f7825317/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34fec9a2-a544-4181-bf3a-b3e81ea7929f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/504c278b298d00620d18e633174d51c060a7cb17ff206253943c4dbd1d57a5e4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c481c3b-feae-4510-83c5-0dc08ce51db0/volumes/kubernetes.io~projected/kube-api-access-mrh86 DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes/kubernetes.io~projected/kube-api-access-9fgwl DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b011ba7c-f34c-44e2-9498-5bd998c41b07/volumes/kubernetes.io~projected/kube-api-access-9bdfx DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58efa8b6b6b953c92c4289f1160ba63e37553f2d48af5a0f206aed7a1971ac2e/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d12efdef0c32486f8df86324b241438773d6318337ca920f2cc7b474630fe4b5/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0caff05880f97cc MacAddress:92:91:90:ce:03:c8 Speed:10000 Mtu:8901} {Name:26bd9805c1799a3 MacAddress:06:55:a4:82:e5:35 Speed:10000 Mtu:8901} {Name:34b6335dff88b63 MacAddress:3e:4f:e7:18:a0:7f Speed:10000 Mtu:8901} {Name:39532e095dc6835 MacAddress:52:02:b6:1d:45:56 Speed:10000 Mtu:8901} {Name:3bd27ddb4aadd0f MacAddress:56:09:9e:c4:a2:05 Speed:10000 Mtu:8901} {Name:3d48388582a3edb MacAddress:da:cc:61:70:85:fe Speed:10000 Mtu:8901} {Name:54475ade28d8777 MacAddress:f2:82:21:df:4f:78 Speed:10000 Mtu:8901} {Name:553d0043bae899b MacAddress:ae:0a:38:12:7b:16 Speed:10000 Mtu:8901} {Name:6c3e87c3c220468 MacAddress:22:c4:42:e5:94:ae Speed:10000 Mtu:8901} {Name:b9efa8322afd0ea MacAddress:da:c0:53:f4:ee:eb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f8:a3:2e:09 Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:c6:e0:81:94:48 Speed:0 Mtu:8901} {Name:c0db4484c8069d4 MacAddress:82:d9:41:0b:54:d4 Speed:10000 Mtu:8901} {Name:d12efdef0c32486 MacAddress:66:ad:ff:09:a7:57 Speed:10000 Mtu:8901} {Name:df1689303ba6d43 MacAddress:2a:d4:40:1f:65:7d Speed:10000 Mtu:8901} {Name:e016e044e470122 MacAddress:d2:ad:12:ed:5c:00 Speed:10000 Mtu:8901} {Name:e93c31328aadee0 MacAddress:22:a3:58:46:ef:c4 Speed:10000 Mtu:8901} {Name:e964e4d9a4ea604 MacAddress:4a:1a:f0:ba:ae:4d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f8:a3:2e:09 Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:af:4c:c1:d9:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.575050 5035 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.575102 5035 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.575629 5035 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.575654 5035 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576006 5035 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576019 5035 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576033 5035 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576052 5035 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576079 5035 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576517 5035 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576870 5035 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576907 5035 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576924 5035 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576937 5035 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576946 5035 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:28:41.577433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.576954 5035 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:28:41.578260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.577605 5035 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:28:41.578260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.577615 5035 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:28:41.578579 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.578565 5035 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:28:41.580694 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.580681 5035 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:28:41.580694 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.580694 5035 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:28:41.580792 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.580783 5035 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 01:28:41.580960 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.580950 5035 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:28:41.580989 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.580980 5035 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:28:41.581067 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581058 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:28:41.581102 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581072 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:28:41.581102 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581078 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:28:41.581102 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581085 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:28:41.581102 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581091 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:28:41.581102 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581097 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:28:41.581102 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581102 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:28:41.581301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581108 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:28:41.581301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581114 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:28:41.581301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581120 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:28:41.581301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581128 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:28:41.581301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581136 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:28:41.581301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581166 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:28:41.581301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.581173 5035 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:28:41.584501 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.584485 5035 server.go:1263] "Started kubelet" Oct 30 01:28:41.584594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.584557 5035 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.584669 5035 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.584730 5035 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.585098 5035 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.586394 5035 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.587827 5035 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.587960 5035 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.587975 5035 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.588069 5035 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.588558 5035 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.588603 5035 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:23:03 +0000 UTC" deadline="2028-06-18 03:13:34.330954636 +0000 UTC" Oct 30 01:28:41.589151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.588638 5035 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14329h44m52.742320684s" Oct 30 01:28:41.585015 ip-10-0-142-110 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:41.591775 5035 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-110.ec2.internal\" not found" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593287 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cafa88a-427e-45c7-a706-d56fd5425610" volumeName="kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-binary-copy" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593327 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cafa88a-427e-45c7-a706-d56fd5425610" volumeName="kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593337 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cafa88a-427e-45c7-a706-d56fd5425610" volumeName="kubernetes.io/projected/5cafa88a-427e-45c7-a706-d56fd5425610-kube-api-access-bd9vt" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593351 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c481c3b-feae-4510-83c5-0dc08ce51db0" volumeName="kubernetes.io/secret/7c481c3b-feae-4510-83c5-0dc08ce51db0-serving-cert" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593360 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8012711c-3e84-43ab-9eac-8e2280709e74" volumeName="kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593367 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b011ba7c-f34c-44e2-9498-5bd998c41b07" volumeName="kubernetes.io/projected/b011ba7c-f34c-44e2-9498-5bd998c41b07-kube-api-access-9bdfx" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593378 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2986b483-9c1e-4d59-89d5-01e287807f8a" volumeName="kubernetes.io/projected/2986b483-9c1e-4d59-89d5-01e287807f8a-kube-api-access-6qsv9" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593385 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32c651f8-7083-4335-a937-682d7a665312" volumeName="kubernetes.io/projected/32c651f8-7083-4335-a937-682d7a665312-kube-api-access-97hrx" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593397 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53172623-6cce-4589-91c3-811476ab540d" volumeName="kubernetes.io/projected/53172623-6cce-4589-91c3-811476ab540d-kube-api-access-d9xrd" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593406 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" volumeName="kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593416 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c481c3b-feae-4510-83c5-0dc08ce51db0" volumeName="kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-config" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593424 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e3639d2-bb67-4554-b567-be193af33780" volumeName="kubernetes.io/projected/3e3639d2-bb67-4554-b567-be193af33780-kube-api-access-wvrtd" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593432 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" volumeName="kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593442 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8012711c-3e84-43ab-9eac-8e2280709e74" volumeName="kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593452 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad411a5-bb79-4e0c-abd3-85c5ba535c93" volumeName="kubernetes.io/empty-dir/aad411a5-bb79-4e0c-abd3-85c5ba535c93-tmp-dir" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593462 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf8cbe6-582e-408b-85a9-d969e18b4682" volumeName="kubernetes.io/configmap/acf8cbe6-582e-408b-85a9-d969e18b4682-serviceca" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593469 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca1fbc85-1496-4e79-9bc9-330a8540a39e" volumeName="kubernetes.io/configmap/ca1fbc85-1496-4e79-9bc9-330a8540a39e-telemetry-config" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593479 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08cfb9ab-35cb-4695-8758-71f71f0c34c3" volumeName="kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-script-lib" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593487 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c903b8f-507e-4dec-801e-3213298d7e0a" volumeName="kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-tmp" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593494 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8ac7159-dcc1-46dc-bae0-eae434811dbe" volumeName="kubernetes.io/configmap/a8ac7159-dcc1-46dc-bae0-eae434811dbe-config" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593503 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8ac7159-dcc1-46dc-bae0-eae434811dbe" volumeName="kubernetes.io/secret/a8ac7159-dcc1-46dc-bae0-eae434811dbe-serving-cert" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593511 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf8cbe6-582e-408b-85a9-d969e18b4682" volumeName="kubernetes.io/projected/acf8cbe6-582e-408b-85a9-d969e18b4682-kube-api-access-4ql9j" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593520 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd24e16c-c31b-4af0-b518-cfe8c85937ac" volumeName="kubernetes.io/configmap/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-config" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593545 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08cfb9ab-35cb-4695-8758-71f71f0c34c3" volumeName="kubernetes.io/projected/08cfb9ab-35cb-4695-8758-71f71f0c34c3-kube-api-access-tjdvd" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593556 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a968163-1344-4372-a8b9-bf896d90e6a9" volumeName="kubernetes.io/configmap/6a968163-1344-4372-a8b9-bf896d90e6a9-config" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593569 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c36ce70e-acde-4698-89b9-37d85639a669" volumeName="kubernetes.io/projected/c36ce70e-acde-4698-89b9-37d85639a669-kube-api-access-78x8b" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593576 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdd98c23-760f-487d-b8d2-be0684665b55" volumeName="kubernetes.io/projected/fdd98c23-760f-487d-b8d2-be0684665b55-kube-api-access-8jm4t" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593590 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ab5bdf-56b9-4176-af0e-f41d92760b84" volumeName="kubernetes.io/projected/02ab5bdf-56b9-4176-af0e-f41d92760b84-kube-api-access-f72c7" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593605 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44ef0b6d-c29b-4925-b06f-5136ef9aa906" volumeName="kubernetes.io/configmap/44ef0b6d-c29b-4925-b06f-5136ef9aa906-konnectivity-ca" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593612 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e60155d-a79a-453f-a17a-902840cdeb14" volumeName="kubernetes.io/projected/6e60155d-a79a-453f-a17a-902840cdeb14-kube-api-access-c7p77" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593621 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" volumeName="kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593631 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad411a5-bb79-4e0c-abd3-85c5ba535c93" volumeName="kubernetes.io/configmap/aad411a5-bb79-4e0c-abd3-85c5ba535c93-config-volume" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593639 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08cfb9ab-35cb-4695-8758-71f71f0c34c3" volumeName="kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-config" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593657 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118aebf2-95cc-4c0a-8f6d-fed0c906316d" volumeName="kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593743 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c903b8f-507e-4dec-801e-3213298d7e0a" volumeName="kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-snapshots" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593751 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cafa88a-427e-45c7-a706-d56fd5425610" volumeName="kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593761 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" volumeName="kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593769 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8012711c-3e84-43ab-9eac-8e2280709e74" volumeName="kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593780 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca1fbc85-1496-4e79-9bc9-330a8540a39e" volumeName="kubernetes.io/projected/ca1fbc85-1496-4e79-9bc9-330a8540a39e-kube-api-access-rtlvp" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593788 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10d07cdc-86b5-4181-9b95-9cfb0341db09" volumeName="kubernetes.io/projected/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-api-access-4hrqt" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593795 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c903b8f-507e-4dec-801e-3213298d7e0a" volumeName="kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593805 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" volumeName="kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593827 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8012711c-3e84-43ab-9eac-8e2280709e74" volumeName="kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593836 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8ac7159-dcc1-46dc-bae0-eae434811dbe" volumeName="kubernetes.io/projected/a8ac7159-dcc1-46dc-bae0-eae434811dbe-kube-api-access-gp748" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593845 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118aebf2-95cc-4c0a-8f6d-fed0c906316d" volumeName="kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593853 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2160cefb-5ed3-4ee9-abc4-9184f7825317" volumeName="kubernetes.io/projected/2160cefb-5ed3-4ee9-abc4-9184f7825317-kube-api-access-q2qx7" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593863 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c481c3b-feae-4510-83c5-0dc08ce51db0" volumeName="kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-trusted-ca" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593870 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b011ba7c-f34c-44e2-9498-5bd998c41b07" volumeName="kubernetes.io/configmap/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-cabundle" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593879 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b82d0410-941f-43e4-9ae9-5d37f9c409be" volumeName="kubernetes.io/projected/b82d0410-941f-43e4-9ae9-5d37f9c409be-kube-api-access-79xfg" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593886 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ab5bdf-56b9-4176-af0e-f41d92760b84" volumeName="kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-stats-auth" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593894 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ab5bdf-56b9-4176-af0e-f41d92760b84" volumeName="kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-default-certificate" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593904 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22ff8c4e-b07d-417b-b4d3-61560b014b77" volumeName="kubernetes.io/empty-dir/22ff8c4e-b07d-417b-b4d3-61560b014b77-tmp-dir" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593912 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc3b9845-9528-42c3-bdae-dc73fec82643" volumeName="kubernetes.io/configmap/bc3b9845-9528-42c3-bdae-dc73fec82643-nginx-conf" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593922 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2160cefb-5ed3-4ee9-abc4-9184f7825317" volumeName="kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-tuned" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593929 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34fec9a2-a544-4181-bf3a-b3e81ea7929f" volumeName="kubernetes.io/projected/34fec9a2-a544-4181-bf3a-b3e81ea7929f-kube-api-access-6gsmz" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593936 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44ef0b6d-c29b-4925-b06f-5136ef9aa906" volumeName="kubernetes.io/secret/44ef0b6d-c29b-4925-b06f-5136ef9aa906-agent-certs" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593985 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8012711c-3e84-43ab-9eac-8e2280709e74" volumeName="kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.593995 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8012711c-3e84-43ab-9eac-8e2280709e74" volumeName="kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594005 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92" volumeName="kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594014 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b011ba7c-f34c-44e2-9498-5bd998c41b07" volumeName="kubernetes.io/secret/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-key" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594023 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd24e16c-c31b-4af0-b518-cfe8c85937ac" volumeName="kubernetes.io/projected/bd24e16c-c31b-4af0-b518-cfe8c85937ac-kube-api-access-t8m9r" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594032 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118aebf2-95cc-4c0a-8f6d-fed0c906316d" volumeName="kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594039 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c903b8f-507e-4dec-801e-3213298d7e0a" volumeName="kubernetes.io/projected/3c903b8f-507e-4dec-801e-3213298d7e0a-kube-api-access-7zsmr" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594050 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e3639d2-bb67-4554-b567-be193af33780" volumeName="kubernetes.io/configmap/3e3639d2-bb67-4554-b567-be193af33780-iptables-alerter-script" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594057 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53172623-6cce-4589-91c3-811476ab540d" volumeName="kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-cni-binary-copy" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594064 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c481c3b-feae-4510-83c5-0dc08ce51db0" volumeName="kubernetes.io/projected/7c481c3b-feae-4510-83c5-0dc08ce51db0-kube-api-access-mrh86" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594074 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c903b8f-507e-4dec-801e-3213298d7e0a" volumeName="kubernetes.io/secret/3c903b8f-507e-4dec-801e-3213298d7e0a-serving-cert" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594081 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53abc737-fbef-4574-96b8-b9adee4fade2" volumeName="kubernetes.io/projected/53abc737-fbef-4574-96b8-b9adee4fade2-kube-api-access-7ntm6" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594091 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a968163-1344-4372-a8b9-bf896d90e6a9" volumeName="kubernetes.io/projected/6a968163-1344-4372-a8b9-bf896d90e6a9-kube-api-access-x9xkz" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594099 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08cfb9ab-35cb-4695-8758-71f71f0c34c3" volumeName="kubernetes.io/secret/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594106 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10d07cdc-86b5-4181-9b95-9cfb0341db09" volumeName="kubernetes.io/empty-dir/10d07cdc-86b5-4181-9b95-9cfb0341db09-data-volume" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594115 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1989a7c0-f591-413a-b111-266ad115c63a" volumeName="kubernetes.io/projected/1989a7c0-f591-413a-b111-266ad115c63a-kube-api-access-8hmfr" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594123 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22ff8c4e-b07d-417b-b4d3-61560b014b77" volumeName="kubernetes.io/projected/22ff8c4e-b07d-417b-b4d3-61560b014b77-kube-api-access-ddck4" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594132 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34fec9a2-a544-4181-bf3a-b3e81ea7929f" volumeName="kubernetes.io/secret/34fec9a2-a544-4181-bf3a-b3e81ea7929f-cert" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594139 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53172623-6cce-4589-91c3-811476ab540d" volumeName="kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-multus-daemon-config" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594188 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" volumeName="kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594198 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad411a5-bb79-4e0c-abd3-85c5ba535c93" volumeName="kubernetes.io/projected/aad411a5-bb79-4e0c-abd3-85c5ba535c93-kube-api-access-bx95x" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594206 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118aebf2-95cc-4c0a-8f6d-fed0c906316d" volumeName="kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594242 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2160cefb-5ed3-4ee9-abc4-9184f7825317" volumeName="kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-tmp" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594249 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a968163-1344-4372-a8b9-bf896d90e6a9" volumeName="kubernetes.io/secret/6a968163-1344-4372-a8b9-bf896d90e6a9-serving-cert" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594262 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" volumeName="kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594310 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8012711c-3e84-43ab-9eac-8e2280709e74" volumeName="kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594322 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971584c9-502d-4dfa-beae-8d8d7ae88cb3" volumeName="kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594331 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eadbf8a4-af76-4fab-b804-13e1ac18afcc" volumeName="kubernetes.io/projected/eadbf8a4-af76-4fab-b804-13e1ac18afcc-kube-api-access-f2hb2" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594371 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08cfb9ab-35cb-4695-8758-71f71f0c34c3" volumeName="kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-env-overrides" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594378 5035 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c903b8f-507e-4dec-801e-3213298d7e0a" volumeName="kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-service-ca-bundle" seLinuxMountContext="" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594385 5035 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:28:41.594613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.594393 5035 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:28:41.601240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.601163 5035 factory.go:55] Registering systemd factory Oct 30 01:28:41.601240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.601185 5035 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:28:41.601758 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.601742 5035 factory.go:153] Registering CRI-O factory Oct 30 01:28:41.601830 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.601761 5035 factory.go:223] Registration of the crio container factory successfully Oct 30 01:28:41.601830 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.601813 5035 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:28:41.601830 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.601831 5035 factory.go:103] Registering Raw factory Oct 30 01:28:41.601973 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.601845 5035 manager.go:1199] Started watching for new ooms in manager Oct 30 01:28:41.602163 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.602151 5035 manager.go:319] Starting recovery of all containers Oct 30 01:28:41.621615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.621590 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/1.log" Oct 30 01:28:41.621904 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.621882 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/1.log" Oct 30 01:28:41.631740 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.631708 5035 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:28:41.634470 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.634440 5035 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:28:41.634470 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.634464 5035 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:28:41.634623 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.634481 5035 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:28:41.634623 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:41.634510 5035 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 01:28:41.644444 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.644423 5035 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:41.644696 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.644676 5035 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 01:28:41.645181 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.645158 5035 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:41.658473 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.658443 5035 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:41.658676 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.658655 5035 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:41.660693 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.660663 5035 generic.go:356] "Generic (PLEG): container finished" podID="573fcfb4d98e5875177cb8f4421e39f7" containerID="d6f09d761f1665b63c0640d2327b81d722520a8b0e0b1ce50369bbcb651c71dc" exitCode=0 Oct 30 01:28:41.668521 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.668468 5035 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="92d28c5aa8b08e4e5bef9518cb06842fa6c8e58105de92459f57a0e1a6707dce" exitCode=0 Oct 30 01:28:41.668521 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.668487 5035 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="3363270155a2a608647c6e7305e74c424b6fc7d92682e0db0709c6ec88697aae" exitCode=0 Oct 30 01:28:41.668521 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.668495 5035 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="e1df9d224c260599eb8bce42bdc3a31e6f46b69b6689eb0fd51adce29ded89ec" exitCode=0 Oct 30 01:28:41.668521 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.668502 5035 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="d7d6d21ae66d78357bcbb0cababff4f090e634e845b319c09f3dea53c52f8948" exitCode=0 Oct 30 01:28:41.668521 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.668509 5035 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="f9b8f0dc558ae76ea19b9b4a64efdb8840f780199e948704af73fb664faf6f41" exitCode=0 Oct 30 01:28:41.668521 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.668516 5035 generic.go:356] "Generic (PLEG): container finished" podID="5cafa88a-427e-45c7-a706-d56fd5425610" containerID="7de466b828952e7010247fc5eb3d8ac1bc9510e85bbf8972bdf51551693c1ce5" exitCode=0 Oct 30 01:28:41.683653 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.683635 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/1.log" Oct 30 01:28:41.683729 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.683659 5035 generic.go:356] "Generic (PLEG): container finished" podID="7c481c3b-feae-4510-83c5-0dc08ce51db0" containerID="d3cbef498b1844773837fd0509bdb11f4bc072e0f4f21fe5b4a4fbd3171d9e31" exitCode=255 Oct 30 01:28:41.698240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.698217 5035 manager.go:324] Recovery completed Oct 30 01:28:41.718778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718754 5035 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:28:41.718778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718774 5035 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:28:41.718886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718793 5035 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:28:41.718936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718902 5035 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 01:28:41.718936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718913 5035 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 01:28:41.718936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718929 5035 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 01:28:41.718936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718934 5035 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 01:28:41.719047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718939 5035 policy_none.go:50] "Start" Oct 30 01:28:41.719047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718948 5035 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:28:41.719047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.718957 5035 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:28:41.719126 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.719069 5035 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 01:28:41.719126 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.719076 5035 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 01:28:41.719126 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.719082 5035 policy_none.go:44] "Start" Oct 30 01:28:41.723291 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.723256 5035 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:28:41.726862 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.726847 5035 manager.go:341] "Starting Device Plugin manager" Oct 30 01:28:41.727348 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:41.726874 5035 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:28:41.727411 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.727351 5035 server.go:87] "Starting device plugin registration server" Oct 30 01:28:41.727606 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.727596 5035 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:28:41.727639 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.727607 5035 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:28:41.727735 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.727717 5035 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:28:41.727906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.727878 5035 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:28:41.728009 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.727994 5035 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:28:41.728009 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.728007 5035 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:28:41.728180 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.728062 5035 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:28:41.727973577Z","UUID":"768d658c-57fc-448e-90f1-49c1b4dc13fa","Handler":null,"Name":"","Endpoint":""} Oct 30 01:28:41.729278 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:41.729256 5035 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:28:41.729416 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.729398 5035 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:28:41.729473 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.729428 5035 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:28:41.734628 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.734590 5035 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal"] Oct 30 01:28:41.735291 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.735273 5035 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.735395 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.735299 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" event={"ID":"573fcfb4d98e5875177cb8f4421e39f7","Type":"ContainerStarted","Data":"e9d20098885e10db0863bb0e4e90211d5fa88aac5bc1d17ed25c8145abc0e53d"} Oct 30 01:28:41.735395 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.735340 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" event={"ID":"573fcfb4d98e5875177cb8f4421e39f7","Type":"ContainerDied","Data":"d6f09d761f1665b63c0640d2327b81d722520a8b0e0b1ce50369bbcb651c71dc"} Oct 30 01:28:41.735395 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.735359 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" event={"ID":"573fcfb4d98e5875177cb8f4421e39f7","Type":"ContainerStarted","Data":"504c278b298d00620d18e633174d51c060a7cb17ff206253943c4dbd1d57a5e4"} Oct 30 01:28:41.735575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.735451 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" event={"ID":"ec075332058530a9d0af488e14c4afa0","Type":"ContainerStarted","Data":"04ccceee214461e3118e6ca32b181d12bd6ca521f219c89bd58b368b3a55c15e"} Oct 30 01:28:41.735575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.735472 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" event={"ID":"ec075332058530a9d0af488e14c4afa0","Type":"ContainerStarted","Data":"a050cd3b4e5df5ce962f0676b2f2386ef0e8184b58a3943f81bc084779c0126d"} Oct 30 01:28:41.735575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.735512 5035 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.772043 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.772023 5035 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:41.772120 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:41.772065 5035 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.772735 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.772720 5035 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:41.772816 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:41.772760 5035 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.828616 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.828593 5035 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:41.829892 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.829875 5035 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:41.829982 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.829907 5035 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:41.829982 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.829920 5035 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-110.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:41.829982 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.829972 5035 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.845458 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.845438 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec075332058530a9d0af488e14c4afa0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" (UID: \"ec075332058530a9d0af488e14c4afa0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.845550 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.845462 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.845550 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.845478 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.873567 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.873546 5035 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.873640 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.873598 5035 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.946653 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.946598 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.946653 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.946624 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec075332058530a9d0af488e14c4afa0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" (UID: \"ec075332058530a9d0af488e14c4afa0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.946653 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.946642 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.946834 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.946683 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.946834 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.946711 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec075332058530a9d0af488e14c4afa0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" (UID: \"ec075332058530a9d0af488e14c4afa0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:41.946834 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:41.946719 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/573fcfb4d98e5875177cb8f4421e39f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" (UID: \"573fcfb4d98e5875177cb8f4421e39f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:42.577364 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.577331 5035 apiserver.go:52] "Watching apiserver" Oct 30 01:28:42.655439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.655372 5035 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:28:42.656291 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.656259 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-8kspl","openshift-insights/periodic-gathering-vvvgp-9njqn","openshift-multus/multus-mf4tj","openshift-dns/node-resolver-bv8wk","openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx","openshift-ovn-kubernetes/ovnkube-node-9w4ql","openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h","openshift-insights/insights-operator-7c4986d449-9659j","openshift-insights/insights-runtime-extractor-hflnn","openshift-multus/network-metrics-daemon-gd5qr","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj","openshift-image-registry/image-registry-7f49dfcbf7-ssq6d","openshift-network-diagnostics/network-check-target-hsc2x","openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4","openshift-console-operator/console-operator-7cd56b8dbc-rzw29","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq","openshift-network-console/networking-console-plugin-86c478bb88-6sdlf","kube-system/global-pull-secret-syncer-xknfd","kube-system/konnectivity-agent-5rdzj","kube-system/kas-connection-checker-685895c7c9-vg24q","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7","openshift-service-ca/service-ca-5fffc97bdf-b9g78","kube-system/kas-connection-checker-685895c7c9-jlj8q","kube-system/kas-connection-checker-685895c7c9-z42rk","openshift-cluster-node-tuning-operator/tuned-6vsqw","openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c","openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc","openshift-ingress/router-default-75f7cfffd5-4l7bn","openshift-multus/multus-additional-cni-plugins-8bdx4","openshift-ingress-canary/ingress-canary-zmb2x","openshift-image-registry/image-registry-6976fcd64d-82clx","openshift-dns/dns-default-dvvl4","openshift-network-operator/iptables-alerter-km427"] Oct 30 01:28:42.661041 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.661016 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.661316 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.661301 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:42.662287 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.662265 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:42.662955 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.662937 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.663608 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.663582 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:42.667405 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.667384 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:42.669185 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.669167 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.670225 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.670209 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.670770 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.670757 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.689831 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.689808 5035 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:28:42.693163 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.693149 5035 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:42.693315 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.693301 5035 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:42.694958 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.694941 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.695000 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.694963 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.695071 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.695058 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.695162 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.695146 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.695195 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.695150 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 01:28:42.696632 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.696618 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 01:28:42.698850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.698835 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 01:28:42.698904 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.698857 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qxp87\"" Oct 30 01:28:42.699013 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.698997 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-sn6bf\"" Oct 30 01:28:42.700673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.700661 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.704559 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704541 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 01:28:42.704623 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704557 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.704623 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704586 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.704623 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704590 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.704623 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704610 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.704790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704628 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:28:42.704790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704636 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.704790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704612 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.704790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704648 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.704790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704667 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 01:28:42.704790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704669 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.704790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704690 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 01:28:42.704790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.704700 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 01:28:42.705085 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.705066 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 01:28:42.708451 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.708435 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sms4c\"" Oct 30 01:28:42.718934 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.718918 5035 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:42.719001 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.718961 5035 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-110.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-110.ec2.internal" Oct 30 01:28:42.722283 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.722268 5035 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:42.722345 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.722305 5035 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-110.ec2.internal" Oct 30 01:28:42.728274 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.728257 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.728327 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.728309 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.728783 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.728760 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 01:28:42.729070 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729048 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.729070 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729062 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.729189 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729073 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.729189 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729049 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.729189 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729109 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.729189 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729126 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.729380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729363 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:28:42.729445 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729411 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.729445 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729419 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.729522 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729495 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:28:42.729522 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729514 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:28:42.729702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729552 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.729742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729733 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:28:42.729789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.729782 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.730398 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730383 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.730484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730400 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:28:42.730484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730447 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.730484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730460 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:28:42.730484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730461 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.730484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730455 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.730673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730488 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 01:28:42.730673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730494 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.730673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730497 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:28:42.730818 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730803 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.730872 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730815 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.730872 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730817 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:28:42.730872 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730841 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:28:42.730872 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730839 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:28:42.731003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730881 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.731003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730886 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.731003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730911 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.731003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730932 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.731003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.730973 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.731155 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.731005 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:28:42.731155 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.731114 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 01:28:42.731247 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.731229 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:42.731328 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.731320 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.732695 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.732682 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 01:28:42.741863 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.741842 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 01:28:42.742333 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.742316 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 01:28:42.749915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.749895 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aad411a5-bb79-4e0c-abd3-85c5ba535c93-config-volume\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.749997 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.749932 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-device-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.749997 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.749955 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-systemd-units\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.749997 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.749982 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx95x\" (UniqueName: \"kubernetes.io/projected/aad411a5-bb79-4e0c-abd3-85c5ba535c93-kube-api-access-bx95x\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.750147 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750046 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8ac7159-dcc1-46dc-bae0-eae434811dbe-config\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:42.750147 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750116 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-bin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.750147 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750143 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aad411a5-bb79-4e0c-abd3-85c5ba535c93-config-volume\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.750280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750201 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-tmp\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.750280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750230 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2qx7\" (UniqueName: \"kubernetes.io/projected/2160cefb-5ed3-4ee9-abc4-9184f7825317-kube-api-access-q2qx7\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.750280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750256 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10d07cdc-86b5-4181-9b95-9cfb0341db09-crio-socket\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.750280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750267 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8ac7159-dcc1-46dc-bae0-eae434811dbe-config\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:42.750452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750284 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-tmp\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.750452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750286 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.750452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750320 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acf8cbe6-582e-408b-85a9-d969e18b4682-serviceca\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:42.750452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750348 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-kubelet-config\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:42.750452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750375 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.750452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750398 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-slash\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.750452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750440 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44ef0b6d-c29b-4925-b06f-5136ef9aa906-agent-certs\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750486 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750517 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bdfx\" (UniqueName: \"kubernetes.io/projected/b011ba7c-f34c-44e2-9498-5bd998c41b07-kube-api-access-9bdfx\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750563 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysconfig\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750569 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750587 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-conf\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750572 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acf8cbe6-582e-408b-85a9-d969e18b4682-serviceca\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750630 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750676 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddck4\" (UniqueName: \"kubernetes.io/projected/22ff8c4e-b07d-417b-b4d3-61560b014b77-kube-api-access-ddck4\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750703 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34fec9a2-a544-4181-bf3a-b3e81ea7929f-cert\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750729 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-conf-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.750773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750750 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e3639d2-bb67-4554-b567-be193af33780-host-slash\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750804 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5xd\" (UniqueName: \"kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750910 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750939 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrqt\" (UniqueName: \"kubernetes.io/projected/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-api-access-4hrqt\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750965 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.750992 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751035 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-kubelet\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751062 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751091 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751145 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751174 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751198 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.751264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751228 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-default-certificate\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751283 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751315 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-multus-daemon-config\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751342 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72c7\" (UniqueName: \"kubernetes.io/projected/02ab5bdf-56b9-4176-af0e-f41d92760b84-kube-api-access-f72c7\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751370 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8ac7159-dcc1-46dc-bae0-eae434811dbe-serving-cert\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751398 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751426 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751450 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751461 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751478 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-etc-kubernetes\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751523 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751593 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-multus-daemon-config\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751595 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qsv9\" (UniqueName: \"kubernetes.io/projected/2986b483-9c1e-4d59-89d5-01e287807f8a-kube-api-access-6qsv9\") pod \"volume-data-source-validator-58c9f5f46d-srr2t\" (UID: \"2986b483-9c1e-4d59-89d5-01e287807f8a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751629 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751676 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-run\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.751810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751781 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10d07cdc-86b5-4181-9b95-9cfb0341db09-data-volume\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751823 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-cnibin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751853 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-key\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751863 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10d07cdc-86b5-4181-9b95-9cfb0341db09-data-volume\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751892 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-sys\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751950 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-node-log\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.751977 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-env-overrides\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752031 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-cnibin\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752129 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-netns\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752168 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-env-overrides\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752179 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-systemd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752209 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a968163-1344-4372-a8b9-bf896d90e6a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752233 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.752286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752259 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.752739 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752292 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-etc-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.752739 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752333 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a968163-1344-4372-a8b9-bf896d90e6a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:42.752739 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752622 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-netd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.752739 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752668 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78x8b\" (UniqueName: \"kubernetes.io/projected/c36ce70e-acde-4698-89b9-37d85639a669-kube-api-access-78x8b\") pod \"kas-connection-checker-685895c7c9-jlj8q\" (UID: \"c36ce70e-acde-4698-89b9-37d85639a669\") " pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" Oct 30 01:28:42.752739 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752706 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntm6\" (UniqueName: \"kubernetes.io/projected/53abc737-fbef-4574-96b8-b9adee4fade2-kube-api-access-7ntm6\") pod \"kas-connection-checker-685895c7c9-vg24q\" (UID: \"53abc737-fbef-4574-96b8-b9adee4fade2\") " pod="kube-system/kas-connection-checker-685895c7c9-vg24q" Oct 30 01:28:42.753021 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752742 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.753021 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752773 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.753021 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752800 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xfg\" (UniqueName: \"kubernetes.io/projected/b82d0410-941f-43e4-9ae9-5d37f9c409be-kube-api-access-79xfg\") pod \"network-check-source-6f78d59f99-jqlwc\" (UID: \"b82d0410-941f-43e4-9ae9-5d37f9c409be\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" Oct 30 01:28:42.753021 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.752907 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.753284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753267 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xrd\" (UniqueName: \"kubernetes.io/projected/53172623-6cce-4589-91c3-811476ab540d-kube-api-access-d9xrd\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.753363 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753308 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.753363 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753339 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmfr\" (UniqueName: \"kubernetes.io/projected/1989a7c0-f591-413a-b111-266ad115c63a-kube-api-access-8hmfr\") pod \"migrator-66df5d496b-5jk7c\" (UID: \"1989a7c0-f591-413a-b111-266ad115c63a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" Oct 30 01:28:42.753469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753372 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdvd\" (UniqueName: \"kubernetes.io/projected/08cfb9ab-35cb-4695-8758-71f71f0c34c3-kube-api-access-tjdvd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.753469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753411 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-snapshots\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.753469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753444 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:42.753636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753482 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-bin\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.753636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753513 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fgwl\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.753636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753576 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.753636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753613 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.753816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753649 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrtd\" (UniqueName: \"kubernetes.io/projected/3e3639d2-bb67-4554-b567-be193af33780-kube-api-access-wvrtd\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:42.753816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753683 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ql9j\" (UniqueName: \"kubernetes.io/projected/acf8cbe6-582e-408b-85a9-d969e18b4682-kube-api-access-4ql9j\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:42.753816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753712 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-cabundle\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:42.753816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753744 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-dbus\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:42.753816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753806 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.754088 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753850 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.754088 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753886 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqx8q\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.754088 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.753917 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-log-socket\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.754310 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754291 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-snapshots\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.754385 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754349 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-service-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.754456 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754394 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-cni-binary-copy\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.754456 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754424 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf8cbe6-582e-408b-85a9-d969e18b4682-host\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:42.754605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754462 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hrx\" (UniqueName: \"kubernetes.io/projected/32c651f8-7083-4335-a937-682d7a665312-kube-api-access-97hrx\") pod \"migrator-66df5d496b-fdcg4\" (UID: \"32c651f8-7083-4335-a937-682d7a665312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" Oct 30 01:28:42.754605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754496 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.754605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754544 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22ff8c4e-b07d-417b-b4d3-61560b014b77-hosts-file\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:42.754605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754581 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gsmz\" (UniqueName: \"kubernetes.io/projected/34fec9a2-a544-4181-bf3a-b3e81ea7929f-kube-api-access-6gsmz\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:42.754605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754610 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-modprobe-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.754900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754810 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-systemd\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.754900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754858 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9xkz\" (UniqueName: \"kubernetes.io/projected/6a968163-1344-4372-a8b9-bf896d90e6a9-kube-api-access-x9xkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:42.754900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754893 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ff8c4e-b07d-417b-b4d3-61560b014b77-tmp-dir\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:42.755111 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754926 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-tmp\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.755111 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754960 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7p77\" (UniqueName: \"kubernetes.io/projected/6e60155d-a79a-453f-a17a-902840cdeb14-kube-api-access-c7p77\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.755111 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.754987 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.755111 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755018 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-socket-dir-parent\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.755111 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755051 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44ef0b6d-c29b-4925-b06f-5136ef9aa906-konnectivity-ca\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:42.755111 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755082 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.755111 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755108 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.755519 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755142 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.755519 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755174 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovn-node-metrics-cert\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.755519 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755209 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zsmr\" (UniqueName: \"kubernetes.io/projected/3c903b8f-507e-4dec-801e-3213298d7e0a-kube-api-access-7zsmr\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.755519 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755242 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc3b9845-9528-42c3-bdae-dc73fec82643-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:42.755519 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755492 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:42.755883 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755774 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-socket-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.755883 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755808 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-tuned\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.755883 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755841 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-config\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.756079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755886 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtlvp\" (UniqueName: \"kubernetes.io/projected/ca1fbc85-1496-4e79-9bc9-330a8540a39e-kube-api-access-rtlvp\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:42.756079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755917 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.756079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755938 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-cabundle\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:42.756079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755948 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.756079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755977 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.756079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756011 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.756079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756041 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-system-cni-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.756079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756073 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c481c3b-feae-4510-83c5-0dc08ce51db0-serving-cert\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756082 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756105 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e3639d2-bb67-4554-b567-be193af33780-iptables-alerter-script\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756145 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-lib-modules\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756177 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca1fbc85-1496-4e79-9bc9-330a8540a39e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756209 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756240 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-k8s-cni-cncf-io\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756210 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c903b8f-507e-4dec-801e-3213298d7e0a-service-ca-bundle\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.755776 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc3b9845-9528-42c3-bdae-dc73fec82643-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756268 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53172623-6cce-4589-91c3-811476ab540d-cni-binary-copy\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756393 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-kubernetes\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756423 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-binary-copy\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756456 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-var-lib-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756490 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-ovn\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756540 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9vt\" (UniqueName: \"kubernetes.io/projected/5cafa88a-427e-45c7-a706-d56fd5425610-kube-api-access-bd9vt\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756569 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad411a5-bb79-4e0c-abd3-85c5ba535c93-tmp-dir\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756586 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.756601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756602 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrh86\" (UniqueName: \"kubernetes.io/projected/7c481c3b-feae-4510-83c5-0dc08ce51db0-kube-api-access-mrh86\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756637 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756673 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756707 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp748\" (UniqueName: \"kubernetes.io/projected/a8ac7159-dcc1-46dc-bae0-eae434811dbe-kube-api-access-gp748\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756724 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ff8c4e-b07d-417b-b4d3-61560b014b77-tmp-dir\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756740 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-registration-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756919 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c481c3b-feae-4510-83c5-0dc08ce51db0-serving-cert\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.756965 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-kubelet\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757005 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jm4t\" (UniqueName: \"kubernetes.io/projected/fdd98c23-760f-487d-b8d2-be0684665b55-kube-api-access-8jm4t\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757003 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e3639d2-bb67-4554-b567-be193af33780-iptables-alerter-script\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757102 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-tuned\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757203 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757258 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44ef0b6d-c29b-4925-b06f-5136ef9aa906-konnectivity-ca\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757327 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-config\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757407 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757471 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c903b8f-507e-4dec-801e-3213298d7e0a-tmp\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.757496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757477 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c903b8f-507e-4dec-801e-3213298d7e0a-serving-cert\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757515 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-trusted-ca\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757613 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-sys-fs\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757642 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad411a5-bb79-4e0c-abd3-85c5ba535c93-tmp-dir\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757660 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a968163-1344-4372-a8b9-bf896d90e6a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757699 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757735 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-config\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757772 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-hostroot\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757811 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8m9r\" (UniqueName: \"kubernetes.io/projected/bd24e16c-c31b-4af0-b518-cfe8c85937ac-kube-api-access-t8m9r\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.757856 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-multus\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758012 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cafa88a-427e-45c7-a706-d56fd5425610-cni-binary-copy\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758103 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758169 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-script-lib\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758207 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-system-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.758249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758241 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.758916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758270 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:42.758916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758308 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-stats-auth\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.758916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758314 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca1fbc85-1496-4e79-9bc9-330a8540a39e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:42.758916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758369 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-netns\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.758916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758401 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-host\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.758916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758430 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-multus-certs\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.758916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758655 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hb2\" (UniqueName: \"kubernetes.io/projected/eadbf8a4-af76-4fab-b804-13e1ac18afcc-kube-api-access-f2hb2\") pod \"kas-connection-checker-685895c7c9-z42rk\" (UID: \"eadbf8a4-af76-4fab-b804-13e1ac18afcc\") " pod="kube-system/kas-connection-checker-685895c7c9-z42rk" Oct 30 01:28:42.758916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758691 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-var-lib-kubelet\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.758916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.758826 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovnkube-script-lib\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.759371 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.759008 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-config\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:42.759601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.759461 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c481c3b-feae-4510-83c5-0dc08ce51db0-trusted-ca\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:42.759854 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.759821 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.781432 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.781407 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx95x\" (UniqueName: \"kubernetes.io/projected/aad411a5-bb79-4e0c-abd3-85c5ba535c93-kube-api-access-bx95x\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.781785 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.781770 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xfg\" (UniqueName: \"kubernetes.io/projected/b82d0410-941f-43e4-9ae9-5d37f9c409be-kube-api-access-79xfg\") pod \"network-check-source-6f78d59f99-jqlwc\" (UID: \"b82d0410-941f-43e4-9ae9-5d37f9c409be\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jqlwc" Oct 30 01:28:42.781892 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.781873 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72c7\" (UniqueName: \"kubernetes.io/projected/02ab5bdf-56b9-4176-af0e-f41d92760b84-kube-api-access-f72c7\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.786182 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.786164 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrqt\" (UniqueName: \"kubernetes.io/projected/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-api-access-4hrqt\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.787208 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.787193 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5xd\" (UniqueName: \"kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd\") pod \"periodic-gathering-vvvgp-9njqn\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:28:42.788243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.788227 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2qx7\" (UniqueName: \"kubernetes.io/projected/2160cefb-5ed3-4ee9-abc4-9184f7825317-kube-api-access-q2qx7\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.791389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.791373 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qsv9\" (UniqueName: \"kubernetes.io/projected/2986b483-9c1e-4d59-89d5-01e287807f8a-kube-api-access-6qsv9\") pod \"volume-data-source-validator-58c9f5f46d-srr2t\" (UID: \"2986b483-9c1e-4d59-89d5-01e287807f8a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-srr2t" Oct 30 01:28:42.807143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.807118 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 01:28:42.807391 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.807359 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 01:28:42.807476 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.807461 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:28:42.807685 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.807658 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7lll\"" Oct 30 01:28:42.807917 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.807880 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hp2kr\"" Oct 30 01:28:42.807978 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.807947 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9xkz\" (UniqueName: \"kubernetes.io/projected/6a968163-1344-4372-a8b9-bf896d90e6a9-kube-api-access-x9xkz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:42.808042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.807989 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdvd\" (UniqueName: \"kubernetes.io/projected/08cfb9ab-35cb-4695-8758-71f71f0c34c3-kube-api-access-tjdvd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.808189 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808150 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrh86\" (UniqueName: \"kubernetes.io/projected/7c481c3b-feae-4510-83c5-0dc08ce51db0-kube-api-access-mrh86\") pod \"console-operator-7cd56b8dbc-rzw29\" (UID: \"7c481c3b-feae-4510-83c5-0dc08ce51db0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:42.808189 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808154 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwqn\" (UniqueName: \"kubernetes.io/projected/971584c9-502d-4dfa-beae-8d8d7ae88cb3-kube-api-access-2bwqn\") pod \"network-check-target-hsc2x\" (UID: \"971584c9-502d-4dfa-beae-8d8d7ae88cb3\") " pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:42.808319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808268 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:28:42.808364 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808349 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:28:42.808489 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808475 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:28:42.808656 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808645 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2p848\"" Oct 30 01:28:42.808757 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808742 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:28:42.808793 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808744 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfcpg\"" Oct 30 01:28:42.808907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808896 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 01:28:42.809008 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.808998 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 01:28:42.809219 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.809201 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-stats-auth\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.809334 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.809285 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 01:28:42.809399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.809387 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 01:28:42.809821 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.809808 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 01:28:42.809997 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.809986 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f29g6\"" Oct 30 01:28:42.810274 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810261 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-f7wwf\"" Oct 30 01:28:42.810274 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810269 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jnw2v\"" Oct 30 01:28:42.810409 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810399 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkdgr\"" Oct 30 01:28:42.810483 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810440 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rwcx9\"" Oct 30 01:28:42.810483 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810474 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zsmr\" (UniqueName: \"kubernetes.io/projected/3c903b8f-507e-4dec-801e-3213298d7e0a-kube-api-access-7zsmr\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.810627 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810554 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nc846\"" Oct 30 01:28:42.810627 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810564 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2gfdm\"" Oct 30 01:28:42.810627 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810572 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7p9xn\"" Oct 30 01:28:42.810851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810664 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglfc\"" Oct 30 01:28:42.810851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810664 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:28:42.810851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810665 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lm6wv\"" Oct 30 01:28:42.810851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810744 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfxzn\"" Oct 30 01:28:42.810851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810776 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c6clm\"" Oct 30 01:28:42.810851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810788 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lzbd2\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810901 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34fec9a2-a544-4181-bf3a-b3e81ea7929f-cert\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.810951 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811013 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zqs4x\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811010 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sj9w4\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811046 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hdprk\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811050 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cz8nh\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811075 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811104 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9qrd6\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811176 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811191 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jr8lv\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811197 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 01:28:42.811255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811204 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qfrfj\"" Oct 30 01:28:42.811771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811270 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:28:42.811771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811302 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 01:28:42.811771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811313 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gx9tc\"" Oct 30 01:28:42.811771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811326 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 01:28:42.811771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811473 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-default-certificate\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.811771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811486 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 01:28:42.811771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811668 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8ac7159-dcc1-46dc-bae0-eae434811dbe-serving-cert\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:42.811771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.811706 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hrx\" (UniqueName: \"kubernetes.io/projected/32c651f8-7083-4335-a937-682d7a665312-kube-api-access-97hrx\") pod \"migrator-66df5d496b-fdcg4\" (UID: \"32c651f8-7083-4335-a937-682d7a665312\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fdcg4" Oct 30 01:28:42.812134 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.812119 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ql9j\" (UniqueName: \"kubernetes.io/projected/acf8cbe6-582e-408b-85a9-d969e18b4682-kube-api-access-4ql9j\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:42.812260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.812247 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b011ba7c-f34c-44e2-9498-5bd998c41b07-signing-key\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:42.815009 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.814977 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.815077 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.815028 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.815349 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.815336 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntm6\" (UniqueName: \"kubernetes.io/projected/53abc737-fbef-4574-96b8-b9adee4fade2-kube-api-access-7ntm6\") pod \"kas-connection-checker-685895c7c9-vg24q\" (UID: \"53abc737-fbef-4574-96b8-b9adee4fade2\") " pod="kube-system/kas-connection-checker-685895c7c9-vg24q" Oct 30 01:28:42.816604 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.816586 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.816845 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.816830 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp748\" (UniqueName: \"kubernetes.io/projected/a8ac7159-dcc1-46dc-bae0-eae434811dbe-kube-api-access-gp748\") pod \"service-ca-operator-544d749c8c-84x2h\" (UID: \"a8ac7159-dcc1-46dc-bae0-eae434811dbe\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" Oct 30 01:28:42.816962 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.816950 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.817712 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.817698 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08cfb9ab-35cb-4695-8758-71f71f0c34c3-ovn-node-metrics-cert\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.819036 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.819016 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a968163-1344-4372-a8b9-bf896d90e6a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj\" (UID: \"6a968163-1344-4372-a8b9-bf896d90e6a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" Oct 30 01:28:42.819133 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.819119 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c903b8f-507e-4dec-801e-3213298d7e0a-serving-cert\") pod \"insights-operator-7c4986d449-9659j\" (UID: \"3c903b8f-507e-4dec-801e-3213298d7e0a\") " pod="openshift-insights/insights-operator-7c4986d449-9659j" Oct 30 01:28:42.820915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.820896 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44ef0b6d-c29b-4925-b06f-5136ef9aa906-agent-certs\") pod \"konnectivity-agent-5rdzj\" (UID: \"44ef0b6d-c29b-4925-b06f-5136ef9aa906\") " pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:42.821235 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.821217 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrtd\" (UniqueName: \"kubernetes.io/projected/3e3639d2-bb67-4554-b567-be193af33780-kube-api-access-wvrtd\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:42.821435 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.821418 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-original-pull-secret\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:42.821991 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.821974 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fgwl\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.822631 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.822616 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jm4t\" (UniqueName: \"kubernetes.io/projected/fdd98c23-760f-487d-b8d2-be0684665b55-kube-api-access-8jm4t\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:42.823690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.823638 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8m9r\" (UniqueName: \"kubernetes.io/projected/bd24e16c-c31b-4af0-b518-cfe8c85937ac-kube-api-access-t8m9r\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:42.824016 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.823982 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddck4\" (UniqueName: \"kubernetes.io/projected/22ff8c4e-b07d-417b-b4d3-61560b014b77-kube-api-access-ddck4\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:42.824192 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.824176 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gsmz\" (UniqueName: \"kubernetes.io/projected/34fec9a2-a544-4181-bf3a-b3e81ea7929f-kube-api-access-6gsmz\") pod \"ingress-canary-zmb2x\" (UID: \"34fec9a2-a544-4181-bf3a-b3e81ea7929f\") " pod="openshift-ingress-canary/ingress-canary-zmb2x" Oct 30 01:28:42.824566 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.824548 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqx8q\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.824848 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.824829 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmfr\" (UniqueName: \"kubernetes.io/projected/1989a7c0-f591-413a-b111-266ad115c63a-kube-api-access-8hmfr\") pod \"migrator-66df5d496b-5jk7c\" (UID: \"1989a7c0-f591-413a-b111-266ad115c63a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5jk7c" Oct 30 01:28:42.825126 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.825110 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xrd\" (UniqueName: \"kubernetes.io/projected/53172623-6cce-4589-91c3-811476ab540d-kube-api-access-d9xrd\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.825815 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.825796 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9vt\" (UniqueName: \"kubernetes.io/projected/5cafa88a-427e-45c7-a706-d56fd5425610-kube-api-access-bd9vt\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.826044 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.826023 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtlvp\" (UniqueName: \"kubernetes.io/projected/ca1fbc85-1496-4e79-9bc9-330a8540a39e-kube-api-access-rtlvp\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:42.826576 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.826556 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.826653 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.826616 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7p77\" (UniqueName: \"kubernetes.io/projected/6e60155d-a79a-453f-a17a-902840cdeb14-kube-api-access-c7p77\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.826948 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.826932 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78x8b\" (UniqueName: \"kubernetes.io/projected/c36ce70e-acde-4698-89b9-37d85639a669-kube-api-access-78x8b\") pod \"kas-connection-checker-685895c7c9-jlj8q\" (UID: \"c36ce70e-acde-4698-89b9-37d85639a669\") " pod="kube-system/kas-connection-checker-685895c7c9-jlj8q" Oct 30 01:28:42.828538 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.828512 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hb2\" (UniqueName: \"kubernetes.io/projected/eadbf8a4-af76-4fab-b804-13e1ac18afcc-kube-api-access-f2hb2\") pod \"kas-connection-checker-685895c7c9-z42rk\" (UID: \"eadbf8a4-af76-4fab-b804-13e1ac18afcc\") " pod="kube-system/kas-connection-checker-685895c7c9-z42rk" Oct 30 01:28:42.828608 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.828588 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bdfx\" (UniqueName: \"kubernetes.io/projected/b011ba7c-f34c-44e2-9498-5bd998c41b07-kube-api-access-9bdfx\") pod \"service-ca-5fffc97bdf-b9g78\" (UID: \"b011ba7c-f34c-44e2-9498-5bd998c41b07\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b9g78" Oct 30 01:28:42.829769 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.829754 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.859289 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859270 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-bin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.859375 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859295 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10d07cdc-86b5-4181-9b95-9cfb0341db09-crio-socket\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.859375 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859313 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-kubelet-config\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:42.859375 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859327 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-slash\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.859375 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859369 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-bin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859380 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10d07cdc-86b5-4181-9b95-9cfb0341db09-crio-socket\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859390 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-slash\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859422 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-kubelet-config\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859432 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysconfig\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859459 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-conf\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859483 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859512 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-conf-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859563 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysconfig\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.859597 5035 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859607 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-conf-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.859613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859613 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-conf\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859609 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e3639d2-bb67-4554-b567-be193af33780-host-slash\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.859655 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:43.359641953 +0000 UTC m=+1.862574155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : secret "metrics-daemon-secret" not found Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859568 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e3639d2-bb67-4554-b567-be193af33780-host-slash\") pod \"iptables-alerter-km427\" (UID: \"3e3639d2-bb67-4554-b567-be193af33780\") " pod="openshift-network-operator/iptables-alerter-km427" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859694 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859709 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-kubelet\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859723 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859737 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859755 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859771 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859777 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-kubelet\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859788 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859810 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859841 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.859863 5035 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.859864 5035 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.859882 5035 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f49dfcbf7-ssq6d: secret "image-registry-tls" not found Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.859905 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert podName:bc3b9845-9528-42c3-bdae-dc73fec82643 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:43.359886746 +0000 UTC m=+1.862818952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6sdlf" (UID: "bc3b9845-9528-42c3-bdae-dc73fec82643") : secret "networking-console-plugin-cert" not found Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859920 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-etc-kubernetes\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859866 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-etc-kubernetes\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.859957 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls podName:7895311d-1a28-4a06-b1b6-2abaa8ed7bbc nodeName:}" failed. No retries permitted until 2026-10-30 01:28:43.359941639 +0000 UTC m=+1.862873856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls") pod "image-registry-7f49dfcbf7-ssq6d" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc") : secret "image-registry-tls" not found Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.859986 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-run\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860012 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-cnibin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860033 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-sys\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860042 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-run\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860055 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-node-log\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860074 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-sys\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860080 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-cnibin\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860101 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-cnibin\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860104 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-node-log\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860100 5035 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860082 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-cnibin\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860105 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-netns\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860140 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-run-netns\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860153 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-systemd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860175 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860183 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-systemd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860194 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-etc-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860207 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-netd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860216 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860222 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860242 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-etc-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860255 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-netd\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860276 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860298 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-bin\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.860323 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:43.360307569 +0000 UTC m=+1.863239796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : configmap references non-existent config key: service-ca.crt Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860324 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-cni-bin\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860343 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860370 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-dbus\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860389 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-log-socket\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860452 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf8cbe6-582e-408b-85a9-d969e18b4682-host\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860478 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22ff8c4e-b07d-417b-b4d3-61560b014b77-hosts-file\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860469 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860487 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-log-socket\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860503 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-modprobe-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860551 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-systemd\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860562 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92-dbus\") pod \"global-pull-secret-syncer-xknfd\" (UID: \"a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92\") " pod="kube-system/global-pull-secret-syncer-xknfd" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860580 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860606 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-socket-dir-parent\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860611 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/22ff8c4e-b07d-417b-b4d3-61560b014b77-hosts-file\") pod \"node-resolver-bv8wk\" (UID: \"22ff8c4e-b07d-417b-b4d3-61560b014b77\") " pod="openshift-dns/node-resolver-bv8wk" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860632 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860642 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-systemd\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860661 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-socket-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860709 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-socket-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860610 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-modprobe-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860726 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860769 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.860792 5035 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860794 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-system-cni-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.860802 5035 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6976fcd64d-82clx: secret "image-registry-tls" not found Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860821 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-lib-modules\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860661 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-socket-dir-parent\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.860836 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls podName:8012711c-3e84-43ab-9eac-8e2280709e74 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:43.360822804 +0000 UTC m=+1.863755004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls") pod "image-registry-6976fcd64d-82clx" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74") : secret "image-registry-tls" not found Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860665 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-multus-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860738 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-sysctl-d\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860865 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860871 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-k8s-cni-cncf-io\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860875 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cafa88a-427e-45c7-a706-d56fd5425610-system-cni-dir\") pod \"multus-additional-cni-plugins-8bdx4\" (UID: \"5cafa88a-427e-45c7-a706-d56fd5425610\") " pod="openshift-multus/multus-additional-cni-plugins-8bdx4" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860905 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-kubernetes\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860915 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-k8s-cni-cncf-io\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860922 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-var-lib-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860945 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-ovn\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860962 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860963 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-etc-kubernetes\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860979 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860983 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-var-lib-openvswitch\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861003 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-registration-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861003 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-run-ovn\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861028 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-kubelet\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861054 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-sys-fs\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.861059 5035 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861067 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-registration-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861078 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-hostroot\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:42.861095 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:43.361083319 +0000 UTC m=+1.864015519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : secret "insights-runtime-extractor-tls" not found Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861113 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-kubelet\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861119 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-hostroot\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.861058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861123 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-multus\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861118 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861151 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-system-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861168 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-sys-fs\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861175 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861170 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-var-lib-cni-multus\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861203 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861213 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-system-cni-dir\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861232 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-netns\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861254 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-host\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861279 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-multus-certs\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861302 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-var-lib-kubelet\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861311 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-netns\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861328 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-device-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861352 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-systemd-units\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861352 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/53172623-6cce-4589-91c3-811476ab540d-host-run-multus-certs\") pod \"multus-mf4tj\" (UID: \"53172623-6cce-4589-91c3-811476ab540d\") " pod="openshift-multus/multus-mf4tj" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861388 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-host\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861425 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e60155d-a79a-453f-a17a-902840cdeb14-device-dir\") pod \"aws-ebs-csi-driver-node-7sbjq\" (UID: \"6e60155d-a79a-453f-a17a-902840cdeb14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sbjq" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861435 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08cfb9ab-35cb-4695-8758-71f71f0c34c3-systemd-units\") pod \"ovnkube-node-9w4ql\" (UID: \"08cfb9ab-35cb-4695-8758-71f71f0c34c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861481 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-var-lib-kubelet\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861548 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2160cefb-5ed3-4ee9-abc4-9184f7825317-lib-modules\") pod \"tuned-6vsqw\" (UID: \"2160cefb-5ed3-4ee9-abc4-9184f7825317\") " pod="openshift-cluster-node-tuning-operator/tuned-6vsqw" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.860573 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf8cbe6-582e-408b-85a9-d969e18b4682-host\") pod \"node-ca-8kspl\" (UID: \"acf8cbe6-582e-408b-85a9-d969e18b4682\") " pod="openshift-image-registry/node-ca-8kspl" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.861682 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10d07cdc-86b5-4181-9b95-9cfb0341db09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.862483 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aad411a5-bb79-4e0c-abd3-85c5ba535c93-metrics-tls\") pod \"dns-default-dvvl4\" (UID: \"aad411a5-bb79-4e0c-abd3-85c5ba535c93\") " pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.862810 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd24e16c-c31b-4af0-b518-cfe8c85937ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5kdp7\" (UID: \"bd24e16c-c31b-4af0-b518-cfe8c85937ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.862958 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ab5bdf-56b9-4176-af0e-f41d92760b84-metrics-certs\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:42.863679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.863584 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca1fbc85-1496-4e79-9bc9-330a8540a39e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c4msx\" (UID: \"ca1fbc85-1496-4e79-9bc9-330a8540a39e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:42.960535 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.960502 5035 scope.go:122] "RemoveContainer" containerID="d3cbef498b1844773837fd0509bdb11f4bc072e0f4f21fe5b4a4fbd3171d9e31" Oct 30 01:28:42.961495 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.961477 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" Oct 30 01:28:42.962753 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.962735 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" Oct 30 01:28:42.970917 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:42.970900 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:43.301358 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.301296 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx"] Oct 30 01:28:43.307396 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:43.307363 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1fbc85_1496_4e79_9bc9_330a8540a39e.slice/crio-3557a44abcd4531d122cf565af77c66a3314145edaf3d533dc6b344eda1b2c8c WatchSource:0}: Error finding container 3557a44abcd4531d122cf565af77c66a3314145edaf3d533dc6b344eda1b2c8c: Status 404 returned error can't find the container with id 3557a44abcd4531d122cf565af77c66a3314145edaf3d533dc6b344eda1b2c8c Oct 30 01:28:43.309821 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.309806 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:28:43.341943 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.341912 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7"] Oct 30 01:28:43.346052 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.346031 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dvvl4"] Oct 30 01:28:43.349436 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:43.349415 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad411a5_bb79_4e0c_abd3_85c5ba535c93.slice/crio-c5323e11f82a77cf78b2f4249afd532cd163a85fee1f8b1ab6f0ea943de07f87 WatchSource:0}: Error finding container c5323e11f82a77cf78b2f4249afd532cd163a85fee1f8b1ab6f0ea943de07f87: Status 404 returned error can't find the container with id c5323e11f82a77cf78b2f4249afd532cd163a85fee1f8b1ab6f0ea943de07f87 Oct 30 01:28:43.367865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.367833 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:43.367952 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.367900 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:43.368003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.367958 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:43.368003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.367993 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:43.368086 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.368025 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:43.368086 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.368065 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:43.368205 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:43.368193 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle podName:02ab5bdf-56b9-4176-af0e-f41d92760b84 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:44.368176267 +0000 UTC m=+2.871108485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle") pod "router-default-75f7cfffd5-4l7bn" (UID: "02ab5bdf-56b9-4176-af0e-f41d92760b84") : configmap references non-existent config key: service-ca.crt Oct 30 01:28:43.368347 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:43.368332 5035 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 01:28:43.368405 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:43.368395 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:44.368381912 +0000 UTC m=+2.871314109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : secret "metrics-daemon-secret" not found Oct 30 01:28:43.368503 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:43.368492 5035 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:28:43.368572 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:43.368538 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:44.368514331 +0000 UTC m=+2.871446528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : secret "insights-runtime-extractor-tls" not found Oct 30 01:28:43.368662 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:43.368645 5035 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:28:43.368715 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:43.368698 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert podName:bc3b9845-9528-42c3-bdae-dc73fec82643 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:44.368683234 +0000 UTC m=+2.871615439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6sdlf" (UID: "bc3b9845-9528-42c3-bdae-dc73fec82643") : secret "networking-console-plugin-cert" not found Oct 30 01:28:43.370640 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.370621 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"image-registry-6976fcd64d-82clx\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:43.370802 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.370786 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"image-registry-7f49dfcbf7-ssq6d\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:43.563417 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.563346 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:43.569646 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.569621 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:43.699301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.699281 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/2.log" Oct 30 01:28:43.699756 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.699742 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/1.log" Oct 30 01:28:43.699799 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.699776 5035 generic.go:356] "Generic (PLEG): container finished" podID="7c481c3b-feae-4510-83c5-0dc08ce51db0" containerID="2375a0e33b5b75eaf3fc491cb3c90f4b6d6192843f117d281119af9d41d26cc8" exitCode=255 Oct 30 01:28:43.699872 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.699856 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" event={"ID":"7c481c3b-feae-4510-83c5-0dc08ce51db0","Type":"ContainerDied","Data":"2375a0e33b5b75eaf3fc491cb3c90f4b6d6192843f117d281119af9d41d26cc8"} Oct 30 01:28:43.699924 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.699911 5035 scope.go:122] "RemoveContainer" containerID="d3cbef498b1844773837fd0509bdb11f4bc072e0f4f21fe5b4a4fbd3171d9e31" Oct 30 01:28:43.700278 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.700254 5035 scope.go:122] "RemoveContainer" containerID="2375a0e33b5b75eaf3fc491cb3c90f4b6d6192843f117d281119af9d41d26cc8" Oct 30 01:28:43.700558 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:43.700491 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rzw29_openshift-console-operator(7c481c3b-feae-4510-83c5-0dc08ce51db0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" podUID="7c481c3b-feae-4510-83c5-0dc08ce51db0" Oct 30 01:28:43.701182 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.701161 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" event={"ID":"ca1fbc85-1496-4e79-9bc9-330a8540a39e","Type":"ContainerStarted","Data":"3557a44abcd4531d122cf565af77c66a3314145edaf3d533dc6b344eda1b2c8c"} Oct 30 01:28:43.702385 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.702364 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dvvl4" event={"ID":"aad411a5-bb79-4e0c-abd3-85c5ba535c93","Type":"ContainerStarted","Data":"c5323e11f82a77cf78b2f4249afd532cd163a85fee1f8b1ab6f0ea943de07f87"} Oct 30 01:28:43.703399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.703379 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" event={"ID":"bd24e16c-c31b-4af0-b518-cfe8c85937ac","Type":"ContainerStarted","Data":"c81c8dc5c075ab782773a490467bb24c7c9bfeacee262e5c7a96641b0e3f0aab"} Oct 30 01:28:43.877707 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.877598 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6976fcd64d-82clx"] Oct 30 01:28:43.883483 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:43.883452 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8012711c_3e84_43ab_9eac_8e2280709e74.slice/crio-b28c08430f21dd3853b39a57e77e86e1789219ed5ae7b221e78fa6704d70a240 WatchSource:0}: Error finding container b28c08430f21dd3853b39a57e77e86e1789219ed5ae7b221e78fa6704d70a240: Status 404 returned error can't find the container with id b28c08430f21dd3853b39a57e77e86e1789219ed5ae7b221e78fa6704d70a240 Oct 30 01:28:43.908654 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:43.907721 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f49dfcbf7-ssq6d"] Oct 30 01:28:43.911885 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:43.911856 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7895311d_1a28_4a06_b1b6_2abaa8ed7bbc.slice/crio-2120faa4a528ddb12ad2f64002d3cfa4d03d8f99421ded91a0cf796fff57b6b7 WatchSource:0}: Error finding container 2120faa4a528ddb12ad2f64002d3cfa4d03d8f99421ded91a0cf796fff57b6b7: Status 404 returned error can't find the container with id 2120faa4a528ddb12ad2f64002d3cfa4d03d8f99421ded91a0cf796fff57b6b7 Oct 30 01:28:44.290811 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.290728 5035 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:44.379843 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.379809 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:44.380034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.379889 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:44.380034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.379919 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:44.380034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.379951 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:44.380034 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:44.379986 5035 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:28:44.380034 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:44.380027 5035 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 01:28:44.380292 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:44.380060 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.380039873 +0000 UTC m=+4.882972076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : secret "insights-runtime-extractor-tls" not found Oct 30 01:28:44.380292 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:44.380080 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs podName:fdd98c23-760f-487d-b8d2-be0684665b55 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.380070117 +0000 UTC m=+4.883002322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs") pod "network-metrics-daemon-gd5qr" (UID: "fdd98c23-760f-487d-b8d2-be0684665b55") : secret "metrics-daemon-secret" not found Oct 30 01:28:44.380292 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:44.380113 5035 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:28:44.380292 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:44.380152 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert podName:bc3b9845-9528-42c3-bdae-dc73fec82643 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:46.380136308 +0000 UTC m=+4.883068513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6sdlf" (UID: "bc3b9845-9528-42c3-bdae-dc73fec82643") : secret "networking-console-plugin-cert" not found Oct 30 01:28:44.380622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.380553 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02ab5bdf-56b9-4176-af0e-f41d92760b84-service-ca-bundle\") pod \"router-default-75f7cfffd5-4l7bn\" (UID: \"02ab5bdf-56b9-4176-af0e-f41d92760b84\") " pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:44.462241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.462152 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:44.710784 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.710242 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" event={"ID":"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc","Type":"ContainerStarted","Data":"cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70"} Oct 30 01:28:44.710784 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.710315 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" event={"ID":"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc","Type":"ContainerStarted","Data":"2120faa4a528ddb12ad2f64002d3cfa4d03d8f99421ded91a0cf796fff57b6b7"} Oct 30 01:28:44.712696 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.712664 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" event={"ID":"8012711c-3e84-43ab-9eac-8e2280709e74","Type":"ContainerStarted","Data":"00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0"} Oct 30 01:28:44.712822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.712746 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" event={"ID":"8012711c-3e84-43ab-9eac-8e2280709e74","Type":"ContainerStarted","Data":"b28c08430f21dd3853b39a57e77e86e1789219ed5ae7b221e78fa6704d70a240"} Oct 30 01:28:44.715935 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.715108 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/2.log" Oct 30 01:28:44.715935 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.715408 5035 scope.go:122] "RemoveContainer" containerID="2375a0e33b5b75eaf3fc491cb3c90f4b6d6192843f117d281119af9d41d26cc8" Oct 30 01:28:44.715935 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:44.715604 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rzw29_openshift-console-operator(7c481c3b-feae-4510-83c5-0dc08ce51db0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" podUID="7c481c3b-feae-4510-83c5-0dc08ce51db0" Oct 30 01:28:44.814587 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:44.814553 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75f7cfffd5-4l7bn"] Oct 30 01:28:44.818556 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:44.817465 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ab5bdf_56b9_4176_af0e_f41d92760b84.slice/crio-cfed472fef35e0efcc3fd1bedc8695b56ca270fa3d5b2f6569e3d3f1b4d9c14b WatchSource:0}: Error finding container cfed472fef35e0efcc3fd1bedc8695b56ca270fa3d5b2f6569e3d3f1b4d9c14b: Status 404 returned error can't find the container with id cfed472fef35e0efcc3fd1bedc8695b56ca270fa3d5b2f6569e3d3f1b4d9c14b Oct 30 01:28:45.726705 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.726677 5035 generic.go:356] "Generic (PLEG): container finished" podID="3c903b8f-507e-4dec-801e-3213298d7e0a" containerID="3a10c0fd672b6ab8b07d93f7b29c15ce7079b0d7af9f13bbc1eaf4d9662ede86" exitCode=0 Oct 30 01:28:45.727449 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.726749 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9659j" event={"ID":"3c903b8f-507e-4dec-801e-3213298d7e0a","Type":"ContainerDied","Data":"3a10c0fd672b6ab8b07d93f7b29c15ce7079b0d7af9f13bbc1eaf4d9662ede86"} Oct 30 01:28:45.727449 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.727043 5035 scope.go:122] "RemoveContainer" containerID="3a10c0fd672b6ab8b07d93f7b29c15ce7079b0d7af9f13bbc1eaf4d9662ede86" Oct 30 01:28:45.729974 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.729949 5035 generic.go:356] "Generic (PLEG): container finished" podID="6a968163-1344-4372-a8b9-bf896d90e6a9" containerID="347ea1d8a0df4785c02866a1aa271554539f3f8cdaf782bdfed1304ad273c911" exitCode=0 Oct 30 01:28:45.730079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.730062 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" event={"ID":"6a968163-1344-4372-a8b9-bf896d90e6a9","Type":"ContainerDied","Data":"347ea1d8a0df4785c02866a1aa271554539f3f8cdaf782bdfed1304ad273c911"} Oct 30 01:28:45.730329 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.730314 5035 scope.go:122] "RemoveContainer" containerID="347ea1d8a0df4785c02866a1aa271554539f3f8cdaf782bdfed1304ad273c911" Oct 30 01:28:45.732695 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.732293 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c4msx" event={"ID":"ca1fbc85-1496-4e79-9bc9-330a8540a39e","Type":"ContainerStarted","Data":"be6fd2a857d7f393c4292cb77abfac90dda7f4917da968b96a1c70bcdcbdd337"} Oct 30 01:28:45.735480 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.735295 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dvvl4" event={"ID":"aad411a5-bb79-4e0c-abd3-85c5ba535c93","Type":"ContainerStarted","Data":"d63d00a59a20b54706b1c31c5d7f4fa5760edeeae23364d2d9a8ad7e92512086"} Oct 30 01:28:45.741026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.740681 5035 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:45.741026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.740898 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" event={"ID":"02ab5bdf-56b9-4176-af0e-f41d92760b84","Type":"ContainerStarted","Data":"395ba8d6dbb44adffa6b61f6687c53ae0e112561dbf02b17b36181186f057a86"} Oct 30 01:28:45.741026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.740921 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" event={"ID":"02ab5bdf-56b9-4176-af0e-f41d92760b84","Type":"ContainerStarted","Data":"cfed472fef35e0efcc3fd1bedc8695b56ca270fa3d5b2f6569e3d3f1b4d9c14b"} Oct 30 01:28:45.741647 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.741175 5035 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:45.750360 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:45.750342 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:28:46.044568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.044484 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:46.399971 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.399898 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:46.399971 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.399950 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:46.400144 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.399980 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:46.400144 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:46.400066 5035 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:28:46.400208 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:46.400150 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls podName:10d07cdc-86b5-4181-9b95-9cfb0341db09 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:50.400130034 +0000 UTC m=+8.903062249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hflnn" (UID: "10d07cdc-86b5-4181-9b95-9cfb0341db09") : secret "insights-runtime-extractor-tls" not found Oct 30 01:28:46.402313 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.402283 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd98c23-760f-487d-b8d2-be0684665b55-metrics-certs\") pod \"network-metrics-daemon-gd5qr\" (UID: \"fdd98c23-760f-487d-b8d2-be0684665b55\") " pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:46.402434 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.402314 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc3b9845-9528-42c3-bdae-dc73fec82643-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6sdlf\" (UID: \"bc3b9845-9528-42c3-bdae-dc73fec82643\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:46.463191 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.463162 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:46.465700 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.465680 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:46.485233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.485214 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:46.485628 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.485614 5035 scope.go:122] "RemoveContainer" containerID="2375a0e33b5b75eaf3fc491cb3c90f4b6d6192843f117d281119af9d41d26cc8" Oct 30 01:28:46.485820 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:46.485804 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rzw29_openshift-console-operator(7c481c3b-feae-4510-83c5-0dc08ce51db0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" podUID="7c481c3b-feae-4510-83c5-0dc08ce51db0" Oct 30 01:28:46.564197 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.564171 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" Oct 30 01:28:46.568673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.568656 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gd5qr" Oct 30 01:28:46.745976 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.745943 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dvvl4" event={"ID":"aad411a5-bb79-4e0c-abd3-85c5ba535c93","Type":"ContainerStarted","Data":"bc479e8c0163e910bf4114910c3a6b8289822f0a7ab0574b38e4751d1ae5ea14"} Oct 30 01:28:46.746409 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.746013 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:46.747828 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.747810 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9659j" event={"ID":"3c903b8f-507e-4dec-801e-3213298d7e0a","Type":"ContainerStarted","Data":"5b6f7faa676b3aed1d778255be51fb9e75f4c948aaa62d7bcb3151e569986cd5"} Oct 30 01:28:46.749620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.749598 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pdwkj" event={"ID":"6a968163-1344-4372-a8b9-bf896d90e6a9","Type":"ContainerStarted","Data":"0340a80ae90fc8084b9eb7c14771f5c101b793bae78960c08a4df027dd0a4d82"} Oct 30 01:28:46.751343 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.751320 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" event={"ID":"bd24e16c-c31b-4af0-b518-cfe8c85937ac","Type":"ContainerStarted","Data":"6739e19b9141d4183eb51f0cae2cd48fde623b2ef8e2e0a4292bc1e3ddcbe69f"} Oct 30 01:28:46.751438 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.751361 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" event={"ID":"bd24e16c-c31b-4af0-b518-cfe8c85937ac","Type":"ContainerStarted","Data":"33fa605b4bf6d7dd7871e062bc830161f1354da3be2d738c75fd9ee076ac0819"} Oct 30 01:28:46.852058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.852038 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:28:46.862465 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.862447 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gd5qr"] Oct 30 01:28:46.865426 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:46.865404 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd98c23_760f_487d_b8d2_be0684665b55.slice/crio-3cd78d2108f8dde660e2ccf35d9a48b49917b5863b4304ea1e19a90a54fbaada WatchSource:0}: Error finding container 3cd78d2108f8dde660e2ccf35d9a48b49917b5863b4304ea1e19a90a54fbaada: Status 404 returned error can't find the container with id 3cd78d2108f8dde660e2ccf35d9a48b49917b5863b4304ea1e19a90a54fbaada Oct 30 01:28:46.874224 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:46.874203 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6sdlf"] Oct 30 01:28:46.877368 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:46.877346 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3b9845_9528_42c3_bdae_dc73fec82643.slice/crio-4fb105873f9ac1ccb42a871fb20a38e6e90b900bbb9195934da3d4a82901a67b WatchSource:0}: Error finding container 4fb105873f9ac1ccb42a871fb20a38e6e90b900bbb9195934da3d4a82901a67b: Status 404 returned error can't find the container with id 4fb105873f9ac1ccb42a871fb20a38e6e90b900bbb9195934da3d4a82901a67b Oct 30 01:28:47.435202 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:47.433979 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:47.435202 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:47.434156 5035 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:47.759658 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:47.759575 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gd5qr" event={"ID":"fdd98c23-760f-487d-b8d2-be0684665b55","Type":"ContainerStarted","Data":"3cd78d2108f8dde660e2ccf35d9a48b49917b5863b4304ea1e19a90a54fbaada"} Oct 30 01:28:47.761800 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:47.761769 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" event={"ID":"bc3b9845-9528-42c3-bdae-dc73fec82643","Type":"ContainerStarted","Data":"4fb105873f9ac1ccb42a871fb20a38e6e90b900bbb9195934da3d4a82901a67b"} Oct 30 01:28:47.762848 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:47.762406 5035 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:48.619092 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:48.619061 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5rdzj" Oct 30 01:28:48.766094 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:48.766058 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gd5qr" event={"ID":"fdd98c23-760f-487d-b8d2-be0684665b55","Type":"ContainerStarted","Data":"9502de396dab0bd3f6ae19154d867a526cfa079adbbef35773e617849075f4df"} Oct 30 01:28:48.766543 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:48.766116 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gd5qr" event={"ID":"fdd98c23-760f-487d-b8d2-be0684665b55","Type":"ContainerStarted","Data":"8de405ce94462e07c2502554514675e68bf2361994dea0b74481310f10f3929a"} Oct 30 01:28:48.767590 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:48.767566 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6sdlf" event={"ID":"bc3b9845-9528-42c3-bdae-dc73fec82643","Type":"ContainerStarted","Data":"1d24e4709cb1d962cb24d27c359266be5f5d8ed6e5f44d870d8c63bb804533a3"} Oct 30 01:28:50.212029 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:50.212002 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9w4ql" Oct 30 01:28:50.433935 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:50.433901 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:50.436195 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:50.436163 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10d07cdc-86b5-4181-9b95-9cfb0341db09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hflnn\" (UID: \"10d07cdc-86b5-4181-9b95-9cfb0341db09\") " pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:50.471353 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:50.471299 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hflnn" Oct 30 01:28:50.763842 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:50.763768 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hflnn"] Oct 30 01:28:50.766819 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:28:50.766787 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d07cdc_86b5_4181_9b95_9cfb0341db09.slice/crio-f8dafef26e899ccfc9bc61c2f51568f4c44beaaf3144a4369d5e7078dac2a01a WatchSource:0}: Error finding container f8dafef26e899ccfc9bc61c2f51568f4c44beaaf3144a4369d5e7078dac2a01a: Status 404 returned error can't find the container with id f8dafef26e899ccfc9bc61c2f51568f4c44beaaf3144a4369d5e7078dac2a01a Oct 30 01:28:50.776333 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:50.776308 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hflnn" event={"ID":"10d07cdc-86b5-4181-9b95-9cfb0341db09","Type":"ContainerStarted","Data":"f8dafef26e899ccfc9bc61c2f51568f4c44beaaf3144a4369d5e7078dac2a01a"} Oct 30 01:28:50.857011 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:50.856985 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:50.857169 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:50.857154 5035 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:50.858366 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:50.858348 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75f7cfffd5-4l7bn" Oct 30 01:28:51.640622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:51.640597 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:28:51.781195 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:51.781123 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hflnn" event={"ID":"10d07cdc-86b5-4181-9b95-9cfb0341db09","Type":"ContainerStarted","Data":"b849e20e14d4df8c57b5ed65d107a23a9a06cc56437c6bdadc6ea4076af90d05"} Oct 30 01:28:51.781195 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:51.781183 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hflnn" event={"ID":"10d07cdc-86b5-4181-9b95-9cfb0341db09","Type":"ContainerStarted","Data":"1f1c1f71347b4cc8e6207d108393374cebccb2feff7629c6fb28597ff7549bf2"} Oct 30 01:28:53.791150 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:53.791108 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hflnn" event={"ID":"10d07cdc-86b5-4181-9b95-9cfb0341db09","Type":"ContainerStarted","Data":"d42c3a10b4aedad6dae43d163b7c0a6474354ac132ffef7048d97c6231febe4f"} Oct 30 01:28:55.635591 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:55.635558 5035 scope.go:122] "RemoveContainer" containerID="2375a0e33b5b75eaf3fc491cb3c90f4b6d6192843f117d281119af9d41d26cc8" Oct 30 01:28:55.800779 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:55.800759 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:28:55.801179 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:55.801162 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/2.log" Oct 30 01:28:55.801276 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:55.801199 5035 generic.go:356] "Generic (PLEG): container finished" podID="7c481c3b-feae-4510-83c5-0dc08ce51db0" containerID="429c915cb4cab4c5cd53e79cefd9bf73a1443cb1cf05d029fc580e7cd7abaf74" exitCode=255 Oct 30 01:28:55.801326 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:55.801273 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" event={"ID":"7c481c3b-feae-4510-83c5-0dc08ce51db0","Type":"ContainerDied","Data":"429c915cb4cab4c5cd53e79cefd9bf73a1443cb1cf05d029fc580e7cd7abaf74"} Oct 30 01:28:55.801384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:55.801334 5035 scope.go:122] "RemoveContainer" containerID="2375a0e33b5b75eaf3fc491cb3c90f4b6d6192843f117d281119af9d41d26cc8" Oct 30 01:28:55.801657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:55.801642 5035 scope.go:122] "RemoveContainer" containerID="429c915cb4cab4c5cd53e79cefd9bf73a1443cb1cf05d029fc580e7cd7abaf74" Oct 30 01:28:55.801846 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:55.801827 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rzw29_openshift-console-operator(7c481c3b-feae-4510-83c5-0dc08ce51db0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" podUID="7c481c3b-feae-4510-83c5-0dc08ce51db0" Oct 30 01:28:56.485697 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:56.485664 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:28:56.764855 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:56.764772 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dvvl4" Oct 30 01:28:56.807693 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:56.807661 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:28:56.808051 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:56.808034 5035 scope.go:122] "RemoveContainer" containerID="429c915cb4cab4c5cd53e79cefd9bf73a1443cb1cf05d029fc580e7cd7abaf74" Oct 30 01:28:56.808559 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:28:56.808226 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rzw29_openshift-console-operator(7c481c3b-feae-4510-83c5-0dc08ce51db0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" podUID="7c481c3b-feae-4510-83c5-0dc08ce51db0" Oct 30 01:28:59.448603 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.448573 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vjh6c-66xss"] Oct 30 01:28:59.454222 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.454196 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.471775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.471748 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h"] Oct 30 01:28:59.474836 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.474815 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" Oct 30 01:28:59.487028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.487003 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vjh6c-66xss"] Oct 30 01:28:59.499054 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.499032 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 01:28:59.504460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.504441 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fgxqf\"" Oct 30 01:28:59.513199 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.513174 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb6ab779-82b5-4347-8847-1ebad5abe3ea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-twz6h\" (UID: \"eb6ab779-82b5-4347-8847-1ebad5abe3ea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" Oct 30 01:28:59.513555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.513516 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6976fcd64d-82clx"] Oct 30 01:28:59.514816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.514793 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h"] Oct 30 01:28:59.520412 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.520383 5035 patch_prober.go:29] interesting pod/image-registry-6976fcd64d-82clx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 01:28:59.520494 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.520437 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" podUID="8012711c-3e84-43ab-9eac-8e2280709e74" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:28:59.593763 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.593734 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-99d5968b4-7dzgs"] Oct 30 01:28:59.597260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.597241 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.614346 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614325 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-trusted-ca-bundle\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.614455 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614353 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/087e4fbf-a966-4c9f-998c-e55b909a7ea8-image-registry-private-configuration\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.614455 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614372 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/087e4fbf-a966-4c9f-998c-e55b909a7ea8-registry-certificates\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.614455 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614389 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-archives-path\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.614634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614452 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-service-ca-bundle\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.614634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614493 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/087e4fbf-a966-4c9f-998c-e55b909a7ea8-bound-sa-token\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.614634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614613 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb6ab779-82b5-4347-8847-1ebad5abe3ea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-twz6h\" (UID: \"eb6ab779-82b5-4347-8847-1ebad5abe3ea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" Oct 30 01:28:59.614780 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614644 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/087e4fbf-a966-4c9f-998c-e55b909a7ea8-trusted-ca\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.614780 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614707 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/087e4fbf-a966-4c9f-998c-e55b909a7ea8-installation-pull-secrets\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.614780 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614745 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/087e4fbf-a966-4c9f-998c-e55b909a7ea8-registry-tls\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.614881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614782 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnfmq\" (UniqueName: \"kubernetes.io/projected/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-kube-api-access-qnfmq\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.614881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614839 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/087e4fbf-a966-4c9f-998c-e55b909a7ea8-ca-trust-extracted\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.614881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.614873 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85fqf\" (UniqueName: \"kubernetes.io/projected/087e4fbf-a966-4c9f-998c-e55b909a7ea8-kube-api-access-85fqf\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.616882 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.616866 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb6ab779-82b5-4347-8847-1ebad5abe3ea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-twz6h\" (UID: \"eb6ab779-82b5-4347-8847-1ebad5abe3ea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" Oct 30 01:28:59.622065 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.622045 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-99d5968b4-7dzgs"] Oct 30 01:28:59.715352 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715324 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/087e4fbf-a966-4c9f-998c-e55b909a7ea8-registry-tls\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.715514 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715365 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfmq\" (UniqueName: \"kubernetes.io/projected/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-kube-api-access-qnfmq\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.715514 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715382 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/087e4fbf-a966-4c9f-998c-e55b909a7ea8-ca-trust-extracted\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.715514 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715408 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85fqf\" (UniqueName: \"kubernetes.io/projected/087e4fbf-a966-4c9f-998c-e55b909a7ea8-kube-api-access-85fqf\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.715514 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715440 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-trusted-ca-bundle\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.715514 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715463 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/087e4fbf-a966-4c9f-998c-e55b909a7ea8-image-registry-private-configuration\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.715514 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715489 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/087e4fbf-a966-4c9f-998c-e55b909a7ea8-registry-certificates\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.715514 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715515 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-archives-path\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.715803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715556 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-service-ca-bundle\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.715803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715577 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/087e4fbf-a966-4c9f-998c-e55b909a7ea8-bound-sa-token\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.715803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715615 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/087e4fbf-a966-4c9f-998c-e55b909a7ea8-trusted-ca\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.715803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715682 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/087e4fbf-a966-4c9f-998c-e55b909a7ea8-installation-pull-secrets\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.716011 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.715852 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/087e4fbf-a966-4c9f-998c-e55b909a7ea8-ca-trust-extracted\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.716082 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.716060 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-archives-path\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.716673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.716648 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/087e4fbf-a966-4c9f-998c-e55b909a7ea8-registry-certificates\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.716811 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.716706 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-service-ca-bundle\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.716945 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.716925 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/087e4fbf-a966-4c9f-998c-e55b909a7ea8-trusted-ca\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.717003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.716947 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-trusted-ca-bundle\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.718166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.718147 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/087e4fbf-a966-4c9f-998c-e55b909a7ea8-installation-pull-secrets\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.718277 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.718260 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/087e4fbf-a966-4c9f-998c-e55b909a7ea8-image-registry-private-configuration\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.718486 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.718466 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/087e4fbf-a966-4c9f-998c-e55b909a7ea8-registry-tls\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.745775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.745754 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/087e4fbf-a966-4c9f-998c-e55b909a7ea8-bound-sa-token\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.746197 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.746177 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85fqf\" (UniqueName: \"kubernetes.io/projected/087e4fbf-a966-4c9f-998c-e55b909a7ea8-kube-api-access-85fqf\") pod \"image-registry-99d5968b4-7dzgs\" (UID: \"087e4fbf-a966-4c9f-998c-e55b909a7ea8\") " pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:28:59.746393 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.746370 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfmq\" (UniqueName: \"kubernetes.io/projected/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-kube-api-access-qnfmq\") pod \"periodic-gathering-vjh6c-66xss\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.768645 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.768624 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:28:59.785936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.785915 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" Oct 30 01:28:59.909242 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:28:59.909218 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:29:00.071147 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.071004 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vjh6c-66xss"] Oct 30 01:29:00.073786 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:29:00.073758 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7f7de3_bb62_4ac9_80f6_a6f626f5a53e.slice/crio-df1e0516a746b9c94fef506a173523dd6f8209a19bf3850d01d28257e93c6545 WatchSource:0}: Error finding container df1e0516a746b9c94fef506a173523dd6f8209a19bf3850d01d28257e93c6545: Status 404 returned error can't find the container with id df1e0516a746b9c94fef506a173523dd6f8209a19bf3850d01d28257e93c6545 Oct 30 01:29:00.093821 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.093797 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h"] Oct 30 01:29:00.101418 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:29:00.101393 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6ab779_82b5_4347_8847_1ebad5abe3ea.slice/crio-f25490c10af6fd8e3604b73f1683fd84e49603c3b4c93531302068f00a62544b WatchSource:0}: Error finding container f25490c10af6fd8e3604b73f1683fd84e49603c3b4c93531302068f00a62544b: Status 404 returned error can't find the container with id f25490c10af6fd8e3604b73f1683fd84e49603c3b4c93531302068f00a62544b Oct 30 01:29:00.205652 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.205622 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-99d5968b4-7dzgs"] Oct 30 01:29:00.208439 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:29:00.208409 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087e4fbf_a966_4c9f_998c_e55b909a7ea8.slice/crio-623149f7a1daebda80a9971edda41fe52b214dfa761ebcd319d6bbb1eb251970 WatchSource:0}: Error finding container 623149f7a1daebda80a9971edda41fe52b214dfa761ebcd319d6bbb1eb251970: Status 404 returned error can't find the container with id 623149f7a1daebda80a9971edda41fe52b214dfa761ebcd319d6bbb1eb251970 Oct 30 01:29:00.835958 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.835915 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vjh6c-66xss" event={"ID":"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e","Type":"ContainerStarted","Data":"7a0bd09b0a240b8116ee6bb6582666614670febf3998d24c9012fee6df1adcce"} Oct 30 01:29:00.836389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.835977 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vjh6c-66xss" event={"ID":"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e","Type":"ContainerStarted","Data":"df1e0516a746b9c94fef506a173523dd6f8209a19bf3850d01d28257e93c6545"} Oct 30 01:29:00.837777 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.837746 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" event={"ID":"eb6ab779-82b5-4347-8847-1ebad5abe3ea","Type":"ContainerStarted","Data":"f25490c10af6fd8e3604b73f1683fd84e49603c3b4c93531302068f00a62544b"} Oct 30 01:29:00.839436 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.839408 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" event={"ID":"087e4fbf-a966-4c9f-998c-e55b909a7ea8","Type":"ContainerStarted","Data":"b2689825d5c9c6cb3f864a2448bc47eeb355df07b56dc765457e120e1ead91ae"} Oct 30 01:29:00.839543 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.839454 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" event={"ID":"087e4fbf-a966-4c9f-998c-e55b909a7ea8","Type":"ContainerStarted","Data":"623149f7a1daebda80a9971edda41fe52b214dfa761ebcd319d6bbb1eb251970"} Oct 30 01:29:00.839543 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.839497 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:29:00.878778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.878717 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vjh6c-66xss" podStartSLOduration=1.8787046109999999 podStartE2EDuration="1.878704611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:00.878498108 +0000 UTC m=+19.381430330" watchObservedRunningTime="2026-10-30 01:29:00.878704611 +0000 UTC m=+19.381636831" Oct 30 01:29:00.926956 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:00.926909 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" podStartSLOduration=1.9268963449999998 podStartE2EDuration="1.926896345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:00.925143998 +0000 UTC m=+19.428076219" watchObservedRunningTime="2026-10-30 01:29:00.926896345 +0000 UTC m=+19.429828556" Oct 30 01:29:01.203376 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:01.203322 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dvvl4_aad411a5-bb79-4e0c-abd3-85c5ba535c93/dns/0.log" Oct 30 01:29:01.343919 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:01.343887 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dvvl4_aad411a5-bb79-4e0c-abd3-85c5ba535c93/kube-rbac-proxy/0.log" Oct 30 01:29:01.843720 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:01.843679 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" event={"ID":"eb6ab779-82b5-4347-8847-1ebad5abe3ea","Type":"ContainerStarted","Data":"ca6c27feeedc01b0f934fcf8f730081ce1a41f0b816392a676949aed9dc1e6f3"} Oct 30 01:29:01.844136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:01.844028 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" Oct 30 01:29:01.848507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:01.848486 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" Oct 30 01:29:01.885959 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:01.885914 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-twz6h" podStartSLOduration=1.830235889 podStartE2EDuration="2.885901434s" totalImagesPullingTime="1.055665545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:01.885432948 +0000 UTC m=+20.388365168" watchObservedRunningTime="2026-10-30 01:29:01.885901434 +0000 UTC m=+20.388833655" Oct 30 01:29:01.945892 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:01.945871 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bv8wk_22ff8c4e-b07d-417b-b4d3-61560b014b77/dns-node-resolver/0.log" Oct 30 01:29:02.144307 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.144230 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6976fcd64d-82clx_8012711c-3e84-43ab-9eac-8e2280709e74/registry/0.log" Oct 30 01:29:02.343429 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.343396 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f49dfcbf7-ssq6d_7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/registry/0.log" Oct 30 01:29:02.543711 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.543684 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-99d5968b4-7dzgs_087e4fbf-a966-4c9f-998c-e55b909a7ea8/registry/0.log" Oct 30 01:29:02.743674 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.743646 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8kspl_acf8cbe6-582e-408b-85a9-d969e18b4682/node-ca/0.log" Oct 30 01:29:02.838015 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.837945 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm"] Oct 30 01:29:02.841391 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.841370 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:02.873011 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.872983 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 01:29:02.873410 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.872996 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:29:02.873410 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.873049 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x2vkp\"" Oct 30 01:29:02.873410 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.873069 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 01:29:02.874640 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.874618 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm"] Oct 30 01:29:02.943285 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.943260 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0904009-0a90-40f2-b917-fecd405d7ec3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:02.943412 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.943321 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0904009-0a90-40f2-b917-fecd405d7ec3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:02.943555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.943523 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4f7j\" (UniqueName: \"kubernetes.io/projected/a0904009-0a90-40f2-b917-fecd405d7ec3-kube-api-access-k4f7j\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:02.943600 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:02.943568 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0904009-0a90-40f2-b917-fecd405d7ec3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.044564 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.044516 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0904009-0a90-40f2-b917-fecd405d7ec3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.044710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.044600 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0904009-0a90-40f2-b917-fecd405d7ec3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.044710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.044631 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4f7j\" (UniqueName: \"kubernetes.io/projected/a0904009-0a90-40f2-b917-fecd405d7ec3-kube-api-access-k4f7j\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.044710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.044658 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0904009-0a90-40f2-b917-fecd405d7ec3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.045313 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.045289 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0904009-0a90-40f2-b917-fecd405d7ec3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.047008 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.046978 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0904009-0a90-40f2-b917-fecd405d7ec3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.047100 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.047022 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0904009-0a90-40f2-b917-fecd405d7ec3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.068937 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.068915 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4f7j\" (UniqueName: \"kubernetes.io/projected/a0904009-0a90-40f2-b917-fecd405d7ec3-kube-api-access-k4f7j\") pod \"prometheus-operator-9cd89b74c-ldmgm\" (UID: \"a0904009-0a90-40f2-b917-fecd405d7ec3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.153794 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.153702 5035 patch_prober.go:29] interesting pod/image-registry-7f49dfcbf7-ssq6d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 01:29:03.153794 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.153774 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" podUID="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:29:03.155766 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.155743 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" Oct 30 01:29:03.343659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.343633 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f7cfffd5-4l7bn_02ab5bdf-56b9-4176-af0e-f41d92760b84/router/0.log" Oct 30 01:29:03.448764 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.448700 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm"] Oct 30 01:29:03.451702 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:29:03.451676 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0904009_0a90_40f2_b917_fecd405d7ec3.slice/crio-dd99a81a72a1d51af44ffd99aa11ff85756099c6b7faf3f54e3f7354c5334f5e WatchSource:0}: Error finding container dd99a81a72a1d51af44ffd99aa11ff85756099c6b7faf3f54e3f7354c5334f5e: Status 404 returned error can't find the container with id dd99a81a72a1d51af44ffd99aa11ff85756099c6b7faf3f54e3f7354c5334f5e Oct 30 01:29:03.543577 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.543551 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zmb2x_34fec9a2-a544-4181-bf3a-b3e81ea7929f/serve-healthcheck-canary/0.log" Oct 30 01:29:03.854508 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:03.854471 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" event={"ID":"a0904009-0a90-40f2-b917-fecd405d7ec3","Type":"ContainerStarted","Data":"dd99a81a72a1d51af44ffd99aa11ff85756099c6b7faf3f54e3f7354c5334f5e"} Oct 30 01:29:04.291375 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:04.291339 5035 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:29:04.291867 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:04.291850 5035 scope.go:122] "RemoveContainer" containerID="429c915cb4cab4c5cd53e79cefd9bf73a1443cb1cf05d029fc580e7cd7abaf74" Oct 30 01:29:04.292117 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:29:04.292095 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rzw29_openshift-console-operator(7c481c3b-feae-4510-83c5-0dc08ce51db0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" podUID="7c481c3b-feae-4510-83c5-0dc08ce51db0" Oct 30 01:29:04.859977 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:04.859943 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" event={"ID":"a0904009-0a90-40f2-b917-fecd405d7ec3","Type":"ContainerStarted","Data":"e02936231b9d05ead633e2f439eaf7aa177fce63268dac1c326daaf89e0ad1eb"} Oct 30 01:29:04.860127 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:04.859997 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" event={"ID":"a0904009-0a90-40f2-b917-fecd405d7ec3","Type":"ContainerStarted","Data":"921f72d5072139a8053ebdc6fc40aa137ace826b91c23adbeb6c8e22bdc145ad"} Oct 30 01:29:04.908397 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:04.908318 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ldmgm" podStartSLOduration=1.741790781 podStartE2EDuration="2.908304724s" totalImagesPullingTime="1.166513943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:04.907199595 +0000 UTC m=+23.410131816" watchObservedRunningTime="2026-10-30 01:29:04.908304724 +0000 UTC m=+23.411236943" Oct 30 01:29:06.857059 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:06.857031 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:29:07.509496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.509462 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7qtpf"] Oct 30 01:29:07.544832 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.544808 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.567432 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.567410 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:29:07.567573 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.567478 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:29:07.567654 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.567578 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tntm7\"" Oct 30 01:29:07.567654 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.567626 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:29:07.584846 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.584826 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-wtmp\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.584947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.584860 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-accelerators-collector-config\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.584947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.584913 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bac59d27-7d89-4f37-89eb-00d81f30223a-root\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.585028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.584965 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bac59d27-7d89-4f37-89eb-00d81f30223a-sys\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.585028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.585002 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-tls\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.585105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.585029 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.585105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.585046 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9dd\" (UniqueName: \"kubernetes.io/projected/bac59d27-7d89-4f37-89eb-00d81f30223a-kube-api-access-cq9dd\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.585105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.585071 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-textfile\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.585196 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.585157 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bac59d27-7d89-4f37-89eb-00d81f30223a-metrics-client-ca\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686357 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bac59d27-7d89-4f37-89eb-00d81f30223a-metrics-client-ca\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686403 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-wtmp\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686517 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-accelerators-collector-config\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686663 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686589 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-wtmp\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686663 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686621 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bac59d27-7d89-4f37-89eb-00d81f30223a-root\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686663 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686649 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bac59d27-7d89-4f37-89eb-00d81f30223a-sys\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686674 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-tls\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686699 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686722 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bac59d27-7d89-4f37-89eb-00d81f30223a-root\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686725 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9dd\" (UniqueName: \"kubernetes.io/projected/bac59d27-7d89-4f37-89eb-00d81f30223a-kube-api-access-cq9dd\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686777 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-textfile\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.686994 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.686724 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bac59d27-7d89-4f37-89eb-00d81f30223a-sys\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.687042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.687002 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bac59d27-7d89-4f37-89eb-00d81f30223a-metrics-client-ca\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.687084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.687070 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-accelerators-collector-config\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.687122 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.687081 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-textfile\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.688957 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.688935 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.689086 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.689066 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bac59d27-7d89-4f37-89eb-00d81f30223a-node-exporter-tls\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.711075 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.711051 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9dd\" (UniqueName: \"kubernetes.io/projected/bac59d27-7d89-4f37-89eb-00d81f30223a-kube-api-access-cq9dd\") pod \"node-exporter-7qtpf\" (UID: \"bac59d27-7d89-4f37-89eb-00d81f30223a\") " pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.909641 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:07.909569 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7qtpf" Oct 30 01:29:07.942085 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:29:07.942054 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac59d27_7d89_4f37_89eb_00d81f30223a.slice/crio-06d2f695cf94f2997931bd6f561be50cd72e5c45e28b41ce89efd0123bf7ec4a WatchSource:0}: Error finding container 06d2f695cf94f2997931bd6f561be50cd72e5c45e28b41ce89efd0123bf7ec4a: Status 404 returned error can't find the container with id 06d2f695cf94f2997931bd6f561be50cd72e5c45e28b41ce89efd0123bf7ec4a Oct 30 01:29:08.877727 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:08.877689 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qtpf" event={"ID":"bac59d27-7d89-4f37-89eb-00d81f30223a","Type":"ContainerStarted","Data":"06d2f695cf94f2997931bd6f561be50cd72e5c45e28b41ce89efd0123bf7ec4a"} Oct 30 01:29:09.519933 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:09.519893 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:29:09.884224 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:09.884177 5035 generic.go:356] "Generic (PLEG): container finished" podID="bac59d27-7d89-4f37-89eb-00d81f30223a" containerID="228f1941ac2730f18f16404e8d1d033d0c6d82a8106c0c6b1bb7c07c105cc358" exitCode=0 Oct 30 01:29:09.884405 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:09.884272 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qtpf" event={"ID":"bac59d27-7d89-4f37-89eb-00d81f30223a","Type":"ContainerDied","Data":"228f1941ac2730f18f16404e8d1d033d0c6d82a8106c0c6b1bb7c07c105cc358"} Oct 30 01:29:10.891399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:10.891363 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qtpf" event={"ID":"bac59d27-7d89-4f37-89eb-00d81f30223a","Type":"ContainerStarted","Data":"b26e54e2c30c3e8024d0a2c168833f0d7a90c6238adfa12ecf68d50e109a0fe5"} Oct 30 01:29:10.891791 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:10.891427 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qtpf" event={"ID":"bac59d27-7d89-4f37-89eb-00d81f30223a","Type":"ContainerStarted","Data":"a53f772e5fe54193195de98c3adb7502ed5632c11e5df7165f5f323d968fb6aa"} Oct 30 01:29:10.937936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:10.937891 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7qtpf" podStartSLOduration=2.474637762 podStartE2EDuration="3.937877068s" totalImagesPullingTime="1.385710689s" totalInitContainerRuntime="77.528617ms" isStatefulPod=true podCreationTimestamp="2026-10-30 01:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:10.937089375 +0000 UTC m=+29.440021595" watchObservedRunningTime="2026-10-30 01:29:10.937877068 +0000 UTC m=+29.440809338" Oct 30 01:29:11.643239 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:11.643205 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hsc2x" Oct 30 01:29:17.626423 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:29:17.626354 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:29:19.635171 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:19.635141 5035 scope.go:122] "RemoveContainer" containerID="429c915cb4cab4c5cd53e79cefd9bf73a1443cb1cf05d029fc580e7cd7abaf74" Oct 30 01:29:19.931220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:19.931143 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:29:19.931363 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:19.931275 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" event={"ID":"7c481c3b-feae-4510-83c5-0dc08ce51db0","Type":"ContainerStarted","Data":"a50d381a49b50f2cd0b68b97b1abc28c53a580f5e6a333528375731e9001d54e"} Oct 30 01:29:19.931758 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:19.931597 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:29:20.291790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:20.291764 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rzw29" Oct 30 01:29:21.849692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:21.849662 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-99d5968b4-7dzgs" Oct 30 01:29:21.948591 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:21.948559 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f49dfcbf7-ssq6d"] Oct 30 01:29:24.534998 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.534911 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" podUID="8012711c-3e84-43ab-9eac-8e2280709e74" containerName="registry" containerID="cri-o://00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0" gracePeriod=30 Oct 30 01:29:24.788359 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.788307 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:29:24.939410 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.939379 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca\") pod \"8012711c-3e84-43ab-9eac-8e2280709e74\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " Oct 30 01:29:24.939589 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.939416 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token\") pod \"8012711c-3e84-43ab-9eac-8e2280709e74\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " Oct 30 01:29:24.939589 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.939454 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") pod \"8012711c-3e84-43ab-9eac-8e2280709e74\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " Oct 30 01:29:24.939715 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.939577 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration\") pod \"8012711c-3e84-43ab-9eac-8e2280709e74\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " Oct 30 01:29:24.939715 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.939700 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted\") pod \"8012711c-3e84-43ab-9eac-8e2280709e74\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " Oct 30 01:29:24.939826 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.939744 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets\") pod \"8012711c-3e84-43ab-9eac-8e2280709e74\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " Oct 30 01:29:24.939826 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.939802 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates\") pod \"8012711c-3e84-43ab-9eac-8e2280709e74\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " Oct 30 01:29:24.939925 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.939843 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl\") pod \"8012711c-3e84-43ab-9eac-8e2280709e74\" (UID: \"8012711c-3e84-43ab-9eac-8e2280709e74\") " Oct 30 01:29:24.940218 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.940044 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca" pod "8012711c-3e84-43ab-9eac-8e2280709e74" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:24.940339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.940297 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates" pod "8012711c-3e84-43ab-9eac-8e2280709e74" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:24.942672 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.942641 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration" pod "8012711c-3e84-43ab-9eac-8e2280709e74" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:24.942672 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.942658 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls" pod "8012711c-3e84-43ab-9eac-8e2280709e74" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.942881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.942723 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl" pod "8012711c-3e84-43ab-9eac-8e2280709e74" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74"). InnerVolumeSpecName "kube-api-access-9fgwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.942881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.942759 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token" pod "8012711c-3e84-43ab-9eac-8e2280709e74" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.942985 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.942916 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets" pod "8012711c-3e84-43ab-9eac-8e2280709e74" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:24.946844 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.946819 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted" pod "8012711c-3e84-43ab-9eac-8e2280709e74" (UID: "8012711c-3e84-43ab-9eac-8e2280709e74"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.959152 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.959129 5035 generic.go:356] "Generic (PLEG): container finished" podID="8012711c-3e84-43ab-9eac-8e2280709e74" containerID="00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0" exitCode=0 Oct 30 01:29:24.959256 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.959173 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" Oct 30 01:29:24.959256 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.959192 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" event={"ID":"8012711c-3e84-43ab-9eac-8e2280709e74","Type":"ContainerDied","Data":"00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0"} Oct 30 01:29:24.959256 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.959230 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6976fcd64d-82clx" event={"ID":"8012711c-3e84-43ab-9eac-8e2280709e74","Type":"ContainerDied","Data":"b28c08430f21dd3853b39a57e77e86e1789219ed5ae7b221e78fa6704d70a240"} Oct 30 01:29:24.959256 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.959249 5035 scope.go:122] "RemoveContainer" containerID="00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0" Oct 30 01:29:24.968466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.968447 5035 scope.go:122] "RemoveContainer" containerID="00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0" Oct 30 01:29:24.968808 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:29:24.968758 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0\": container with ID starting with 00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0 not found: ID does not exist" containerID="00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0" Oct 30 01:29:24.968905 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:24.968786 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0"} err="failed to get container status \"00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0\": rpc error: code = NotFound desc = could not find container \"00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0\": container with ID starting with 00f520fcc0a900ca13edc1a33b64298dd2a77f25fcc355d80904100fe1aea2a0 not found: ID does not exist" Oct 30 01:29:25.017425 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.017405 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6976fcd64d-82clx"] Oct 30 01:29:25.028728 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.028701 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6976fcd64d-82clx"] Oct 30 01:29:25.040850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.040796 5035 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8012711c-3e84-43ab-9eac-8e2280709e74-ca-trust-extracted\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.040850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.040820 5035 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-installation-pull-secrets\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.040850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.040833 5035 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-registry-certificates\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.040850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.040847 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fgwl\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-kube-api-access-9fgwl\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.041036 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.040864 5035 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8012711c-3e84-43ab-9eac-8e2280709e74-trusted-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.041036 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.040877 5035 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-bound-sa-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.041036 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.040889 5035 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8012711c-3e84-43ab-9eac-8e2280709e74-registry-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.041036 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.040901 5035 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8012711c-3e84-43ab-9eac-8e2280709e74-image-registry-private-configuration\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.640134 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:25.640102 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8012711c-3e84-43ab-9eac-8e2280709e74" path="/var/lib/kubelet/pods/8012711c-3e84-43ab-9eac-8e2280709e74/volumes" Oct 30 01:29:31.332026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:31.331990 5035 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5rdzj_44ef0b6d-c29b-4925-b06f-5136ef9aa906/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:31.911934 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:31.911899 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vvvgp-9njqn"] Oct 30 01:29:31.912162 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:31.912134 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" podUID="118aebf2-95cc-4c0a-8f6d-fed0c906316d" containerName="insights-gathering" containerID="cri-o://49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb" gracePeriod=30 Oct 30 01:29:32.156365 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.156341 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:29:32.300140 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.300113 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd\" (UniqueName: \"kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd\") pod \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " Oct 30 01:29:32.300302 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.300154 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle\") pod \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " Oct 30 01:29:32.300302 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.300200 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle\") pod \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " Oct 30 01:29:32.300302 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.300235 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path\") pod \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\" (UID: \"118aebf2-95cc-4c0a-8f6d-fed0c906316d\") " Oct 30 01:29:32.300600 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.300566 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path" pod "118aebf2-95cc-4c0a-8f6d-fed0c906316d" (UID: "118aebf2-95cc-4c0a-8f6d-fed0c906316d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:32.300713 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.300620 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle" pod "118aebf2-95cc-4c0a-8f6d-fed0c906316d" (UID: "118aebf2-95cc-4c0a-8f6d-fed0c906316d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:32.300713 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.300629 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle" pod "118aebf2-95cc-4c0a-8f6d-fed0c906316d" (UID: "118aebf2-95cc-4c0a-8f6d-fed0c906316d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:32.302468 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.302448 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd" pod "118aebf2-95cc-4c0a-8f6d-fed0c906316d" (UID: "118aebf2-95cc-4c0a-8f6d-fed0c906316d"). InnerVolumeSpecName "kube-api-access-9c5xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:32.401466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.401426 5035 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-trusted-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:32.401466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.401462 5035 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/118aebf2-95cc-4c0a-8f6d-fed0c906316d-archives-path\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:32.401466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.401471 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c5xd\" (UniqueName: \"kubernetes.io/projected/118aebf2-95cc-4c0a-8f6d-fed0c906316d-kube-api-access-9c5xd\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:32.401871 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.401502 5035 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118aebf2-95cc-4c0a-8f6d-fed0c906316d-service-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:32.996609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.996573 5035 generic.go:356] "Generic (PLEG): container finished" podID="118aebf2-95cc-4c0a-8f6d-fed0c906316d" containerID="49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb" exitCode=2 Oct 30 01:29:32.996796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.996619 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" Oct 30 01:29:32.996796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.996624 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" event={"ID":"118aebf2-95cc-4c0a-8f6d-fed0c906316d","Type":"ContainerDied","Data":"49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb"} Oct 30 01:29:32.996796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.996723 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vvvgp-9njqn" event={"ID":"118aebf2-95cc-4c0a-8f6d-fed0c906316d","Type":"ContainerDied","Data":"d12efdef0c32486f8df86324b241438773d6318337ca920f2cc7b474630fe4b5"} Oct 30 01:29:32.996796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:32.996744 5035 scope.go:122] "RemoveContainer" containerID="49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb" Oct 30 01:29:33.005599 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:33.005575 5035 scope.go:122] "RemoveContainer" containerID="49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb" Oct 30 01:29:33.005857 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:29:33.005839 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb\": container with ID starting with 49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb not found: ID does not exist" containerID="49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb" Oct 30 01:29:33.005926 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:33.005869 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb"} err="failed to get container status \"49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb\": rpc error: code = NotFound desc = could not find container \"49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb\": container with ID starting with 49b6eedb67bca5a5011c5630fb217fb29423cd6ed257463bcd8c3e4d05485cdb not found: ID does not exist" Oct 30 01:29:33.054650 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:33.054607 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vvvgp-9njqn"] Oct 30 01:29:33.057198 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:33.057176 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-vvvgp-9njqn"] Oct 30 01:29:33.640462 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:33.640430 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118aebf2-95cc-4c0a-8f6d-fed0c906316d" path="/var/lib/kubelet/pods/118aebf2-95cc-4c0a-8f6d-fed0c906316d/volumes" Oct 30 01:29:46.970766 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:46.970707 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" podUID="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" containerName="registry" containerID="cri-o://cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70" gracePeriod=30 Oct 30 01:29:47.071863 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.071836 5035 generic.go:356] "Generic (PLEG): container finished" podID="a8ac7159-dcc1-46dc-bae0-eae434811dbe" containerID="3a9844d501e478e73a0af99607fb9948506f820dde7ea2400d2c47d9c90d6d22" exitCode=0 Oct 30 01:29:47.071973 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.071910 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" event={"ID":"a8ac7159-dcc1-46dc-bae0-eae434811dbe","Type":"ContainerDied","Data":"3a9844d501e478e73a0af99607fb9948506f820dde7ea2400d2c47d9c90d6d22"} Oct 30 01:29:47.072222 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.072208 5035 scope.go:122] "RemoveContainer" containerID="3a9844d501e478e73a0af99607fb9948506f820dde7ea2400d2c47d9c90d6d22" Oct 30 01:29:47.238803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.238781 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:29:47.327025 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.326991 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") pod \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " Oct 30 01:29:47.327175 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327030 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates\") pod \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " Oct 30 01:29:47.327175 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327065 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q\") pod \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " Oct 30 01:29:47.327284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327231 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets\") pod \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " Oct 30 01:29:47.327284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327277 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca\") pod \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " Oct 30 01:29:47.327378 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327311 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted\") pod \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " Oct 30 01:29:47.327378 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327349 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration\") pod \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " Oct 30 01:29:47.327475 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327406 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token\") pod \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\" (UID: \"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc\") " Oct 30 01:29:47.327475 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327446 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates" pod "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:47.327659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327640 5035 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-certificates\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:47.328060 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.327979 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca" pod "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:47.329717 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.329681 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets" pod "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:47.329717 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.329702 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls" pod "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:47.329898 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.329878 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q" pod "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc"). InnerVolumeSpecName "kube-api-access-qqx8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:47.329994 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.329968 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration" pod "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:47.330305 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.330276 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token" pod "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:47.335277 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.335252 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted" pod "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" (UID: "7895311d-1a28-4a06-b1b6-2abaa8ed7bbc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:47.428692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.428663 5035 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-bound-sa-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:47.428692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.428692 5035 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-registry-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:47.428824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.428707 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqx8q\" (UniqueName: \"kubernetes.io/projected/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-kube-api-access-qqx8q\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:47.428824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.428723 5035 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-installation-pull-secrets\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:47.428824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.428737 5035 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-trusted-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:47.428824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.428750 5035 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-ca-trust-extracted\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:47.428824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:47.428766 5035 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc-image-registry-private-configuration\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:29:47.627392 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:29:47.627313 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:29:48.078009 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.077965 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-84x2h" event={"ID":"a8ac7159-dcc1-46dc-bae0-eae434811dbe","Type":"ContainerStarted","Data":"c769f3919d29b06936607014e1c9654be4eb2a347ee701b46130c0aef7686f24"} Oct 30 01:29:48.079241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.079213 5035 generic.go:356] "Generic (PLEG): container finished" podID="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" containerID="cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70" exitCode=0 Oct 30 01:29:48.079378 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.079250 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" event={"ID":"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc","Type":"ContainerDied","Data":"cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70"} Oct 30 01:29:48.079378 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.079306 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" event={"ID":"7895311d-1a28-4a06-b1b6-2abaa8ed7bbc","Type":"ContainerDied","Data":"2120faa4a528ddb12ad2f64002d3cfa4d03d8f99421ded91a0cf796fff57b6b7"} Oct 30 01:29:48.079378 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.079334 5035 scope.go:122] "RemoveContainer" containerID="cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70" Oct 30 01:29:48.079378 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.079260 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f49dfcbf7-ssq6d" Oct 30 01:29:48.087572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.087554 5035 scope.go:122] "RemoveContainer" containerID="cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70" Oct 30 01:29:48.087882 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:29:48.087849 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70\": container with ID starting with cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70 not found: ID does not exist" containerID="cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70" Oct 30 01:29:48.087973 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.087888 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70"} err="failed to get container status \"cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70\": rpc error: code = NotFound desc = could not find container \"cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70\": container with ID starting with cda7279fc8ec9d6396fd99dcdd50a6003bb86c21e38ac7b4b72fa5fd98b86a70 not found: ID does not exist" Oct 30 01:29:48.160475 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.160448 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f49dfcbf7-ssq6d"] Oct 30 01:29:48.173565 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:48.173538 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f49dfcbf7-ssq6d"] Oct 30 01:29:49.640768 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:29:49.640736 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" path="/var/lib/kubelet/pods/7895311d-1a28-4a06-b1b6-2abaa8ed7bbc/volumes" Oct 30 01:30:17.625803 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:30:17.625762 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:30:25.232232 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:25.232194 5035 generic.go:356] "Generic (PLEG): container finished" podID="3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e" containerID="7a0bd09b0a240b8116ee6bb6582666614670febf3998d24c9012fee6df1adcce" exitCode=0 Oct 30 01:30:25.232656 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:25.232245 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vjh6c-66xss" event={"ID":"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e","Type":"ContainerDied","Data":"7a0bd09b0a240b8116ee6bb6582666614670febf3998d24c9012fee6df1adcce"} Oct 30 01:30:26.370411 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.370388 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:30:26.436683 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.436648 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-kube-api-access-qnfmq\" (UniqueName: \"kubernetes.io/projected/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-kube-api-access-qnfmq\") pod \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " Oct 30 01:30:26.436860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.436704 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-trusted-ca-bundle\") pod \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " Oct 30 01:30:26.436860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.436755 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-service-ca-bundle\") pod \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " Oct 30 01:30:26.436860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.436781 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-archives-path\") pod \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\" (UID: \"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e\") " Oct 30 01:30:26.437143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.437119 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-service-ca-bundle" pod "3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e" (UID: "3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:26.437206 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.437148 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-trusted-ca-bundle" pod "3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e" (UID: "3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:26.437206 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.437189 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-archives-path" pod "3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e" (UID: "3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:26.439010 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.438986 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-kube-api-access-qnfmq" pod "3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e" (UID: "3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e"). InnerVolumeSpecName "kube-api-access-qnfmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:26.542735 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.538612 5035 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-service-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.542735 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.538669 5035 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-archives-path\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.542735 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.538688 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnfmq\" (UniqueName: \"kubernetes.io/projected/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-kube-api-access-qnfmq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.542735 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:26.538705 5035 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e-trusted-ca-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:30:27.241482 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:27.241443 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vjh6c-66xss" event={"ID":"3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e","Type":"ContainerDied","Data":"df1e0516a746b9c94fef506a173523dd6f8209a19bf3850d01d28257e93c6545"} Oct 30 01:30:27.241686 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:27.241498 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df1e0516a746b9c94fef506a173523dd6f8209a19bf3850d01d28257e93c6545" Oct 30 01:30:27.241686 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:30:27.241496 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vjh6c-66xss" Oct 30 01:30:47.626413 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:30:47.626374 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:31:17.626959 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:31:17.626877 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:31:47.625670 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:31:47.625625 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:32:17.627100 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:32:17.627059 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:32:47.626825 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:32:47.626776 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:32:52.923916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.923877 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h"] Oct 30 01:32:52.924427 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924328 5035 cpu_manager.go:414] "containerMap: removing container" podUID="118aebf2-95cc-4c0a-8f6d-fed0c906316d" containerName="insights-gathering" Oct 30 01:32:52.924427 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924347 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118aebf2-95cc-4c0a-8f6d-fed0c906316d" containerName="insights-gathering" Oct 30 01:32:52.924427 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924372 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8012711c-3e84-43ab-9eac-8e2280709e74" containerName="registry" Oct 30 01:32:52.924427 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924381 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8012711c-3e84-43ab-9eac-8e2280709e74" containerName="registry" Oct 30 01:32:52.924427 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924399 5035 cpu_manager.go:414] "containerMap: removing container" podUID="3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e" containerName="insights-gathering" Oct 30 01:32:52.924427 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924408 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e" containerName="insights-gathering" Oct 30 01:32:52.924750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924433 5035 cpu_manager.go:414] "containerMap: removing container" podUID="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" containerName="registry" Oct 30 01:32:52.924750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924441 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" containerName="registry" Oct 30 01:32:52.924750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924544 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8012711c-3e84-43ab-9eac-8e2280709e74" containerName="registry" Oct 30 01:32:52.924750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924558 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7f7de3-bb62-4ac9-80f6-a6f626f5a53e" containerName="insights-gathering" Oct 30 01:32:52.924750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924572 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="7895311d-1a28-4a06-b1b6-2abaa8ed7bbc" containerName="registry" Oct 30 01:32:52.924750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.924584 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="118aebf2-95cc-4c0a-8f6d-fed0c906316d" containerName="insights-gathering" Oct 30 01:32:52.927510 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.927489 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:52.947782 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.947760 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5dmpm\"" Oct 30 01:32:52.961322 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.961300 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 01:32:52.962905 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:52.962875 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h"] Oct 30 01:32:53.040370 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.040308 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.040370 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.040343 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.040494 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.040383 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.040554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.040487 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.040554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.040547 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2j49\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-kube-api-access-l2j49\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.040621 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.040570 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.040621 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.040586 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.040621 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.040608 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.040713 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.040656 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b359b740-709a-4fd9-ade6-4bf08d7b827b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141183 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141154 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141324 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141190 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141324 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141219 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2j49\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-kube-api-access-l2j49\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141324 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141243 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141324 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141268 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141324 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141299 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141321 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b359b740-709a-4fd9-ade6-4bf08d7b827b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141467 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141503 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141626 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141658 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.141742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.141683 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.142200 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.142173 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.142308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.142236 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b359b740-709a-4fd9-ade6-4bf08d7b827b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.143589 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.143568 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.143826 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.143808 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.168096 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.168068 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.169197 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.169168 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2j49\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-kube-api-access-l2j49\") pod \"maas-default-gateway-openshift-default-75d7c785f-8pw9h\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.239976 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.239953 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:53.549237 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.549201 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h"] Oct 30 01:32:53.552365 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:32:53.552335 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb359b740_709a_4fd9_ade6_4bf08d7b827b.slice/crio-f8051db5ff8d1a021279c9c5ddee618f0297afa7c372a44e65a9870e317dca02 WatchSource:0}: Error finding container f8051db5ff8d1a021279c9c5ddee618f0297afa7c372a44e65a9870e317dca02: Status 404 returned error can't find the container with id f8051db5ff8d1a021279c9c5ddee618f0297afa7c372a44e65a9870e317dca02 Oct 30 01:32:53.814691 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:53.814605 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" event={"ID":"b359b740-709a-4fd9-ade6-4bf08d7b827b","Type":"ContainerStarted","Data":"f8051db5ff8d1a021279c9c5ddee618f0297afa7c372a44e65a9870e317dca02"} Oct 30 01:32:56.424355 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.424319 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:56.424763 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.424400 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:56.424763 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.424451 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:56.720100 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.720061 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d"] Oct 30 01:32:56.724513 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.724461 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.742347 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.742209 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d"] Oct 30 01:32:56.754171 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.754148 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 01:32:56.754289 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.754182 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 01:32:56.754289 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.754231 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-hvdh8\"" Oct 30 01:32:56.766453 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.766425 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9"] Oct 30 01:32:56.770650 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.770632 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.789505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.789481 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9"] Oct 30 01:32:56.800140 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.800116 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg"] Oct 30 01:32:56.803577 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.803559 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:56.806192 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.806174 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-kb7b8\"" Oct 30 01:32:56.823577 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.823549 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg"] Oct 30 01:32:56.832934 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.832912 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" event={"ID":"b359b740-709a-4fd9-ade6-4bf08d7b827b","Type":"ContainerStarted","Data":"a466a4de1b11b6eccf402360c9dc56a62dbc58114254ae93d9987a197c4627f3"} Oct 30 01:32:56.845035 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.845014 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-5r7vv\"" Oct 30 01:32:56.871699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.871676 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.871780 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.871716 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgslx\" (UniqueName: \"kubernetes.io/projected/9ac2a367-1394-4333-92ed-d0e3677550ad-kube-api-access-qgslx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:56.871832 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.871805 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:56.871868 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.871835 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.871868 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.871852 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.871933 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.871922 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.871978 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.871949 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:56.871978 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.871971 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8gl\" (UniqueName: \"kubernetes.io/projected/0e5de0aa-5915-4679-b44f-c214cf648d4b-kube-api-access-xd8gl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.872045 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.872016 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjxl\" (UniqueName: \"kubernetes.io/projected/9af8d363-0df7-4e68-af45-a6b1f68d8505-kube-api-access-vtjxl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.913046 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.913006 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" podStartSLOduration=2.043635285 podStartE2EDuration="4.912995229s" totalImagesPullingTime="2.869359944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:56.911950252 +0000 UTC m=+255.414882473" watchObservedRunningTime="2026-10-30 01:32:56.912995229 +0000 UTC m=+255.415927486" Oct 30 01:32:56.972990 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.972964 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:56.973079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.972993 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.973079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973009 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.973079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973035 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.973079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973069 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:56.973252 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973118 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8gl\" (UniqueName: \"kubernetes.io/projected/0e5de0aa-5915-4679-b44f-c214cf648d4b-kube-api-access-xd8gl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.973252 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973161 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjxl\" (UniqueName: \"kubernetes.io/projected/9af8d363-0df7-4e68-af45-a6b1f68d8505-kube-api-access-vtjxl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.973703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973452 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.973703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973480 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:56.973703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973456 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.973703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973455 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:56.973703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973511 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.973703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973522 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.973703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973598 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgslx\" (UniqueName: \"kubernetes.io/projected/9ac2a367-1394-4333-92ed-d0e3677550ad-kube-api-access-qgslx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:56.973991 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.973786 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.997330 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.997273 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8gl\" (UniqueName: \"kubernetes.io/projected/0e5de0aa-5915-4679-b44f-c214cf648d4b-kube-api-access-xd8gl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:56.997330 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.997307 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjxl\" (UniqueName: \"kubernetes.io/projected/9af8d363-0df7-4e68-af45-a6b1f68d8505-kube-api-access-vtjxl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:56.997476 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:56.997362 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgslx\" (UniqueName: \"kubernetes.io/projected/9ac2a367-1394-4333-92ed-d0e3677550ad-kube-api-access-qgslx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:57.040323 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.040296 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:32:57.086736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.086705 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:32:57.119851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.119824 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:32:57.240612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.240578 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:57.244830 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.244806 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:57.341029 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.340912 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d"] Oct 30 01:32:57.343617 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:32:57.343588 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5de0aa_5915_4679_b44f_c214cf648d4b.slice/crio-53b16bb8d4825f67d4ab9e5948121c9fc453567fbf9bbb08b31dc8a2139e7f22 WatchSource:0}: Error finding container 53b16bb8d4825f67d4ab9e5948121c9fc453567fbf9bbb08b31dc8a2139e7f22: Status 404 returned error can't find the container with id 53b16bb8d4825f67d4ab9e5948121c9fc453567fbf9bbb08b31dc8a2139e7f22 Oct 30 01:32:57.392445 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.392422 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9"] Oct 30 01:32:57.394807 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:32:57.394782 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af8d363_0df7_4e68_af45_a6b1f68d8505.slice/crio-60f8cca3477fc303c90273184515b90e107f0a9fb3e2baf961a13b48f0d8bcbc WatchSource:0}: Error finding container 60f8cca3477fc303c90273184515b90e107f0a9fb3e2baf961a13b48f0d8bcbc: Status 404 returned error can't find the container with id 60f8cca3477fc303c90273184515b90e107f0a9fb3e2baf961a13b48f0d8bcbc Oct 30 01:32:57.426337 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.426315 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg"] Oct 30 01:32:57.428641 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:32:57.428619 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac2a367_1394_4333_92ed_d0e3677550ad.slice/crio-b16d685da8c41f39fbf9b34649acc7a8905050725cc0b35de41ded038a9f372d WatchSource:0}: Error finding container b16d685da8c41f39fbf9b34649acc7a8905050725cc0b35de41ded038a9f372d: Status 404 returned error can't find the container with id b16d685da8c41f39fbf9b34649acc7a8905050725cc0b35de41ded038a9f372d Oct 30 01:32:57.840901 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.840859 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" event={"ID":"9af8d363-0df7-4e68-af45-a6b1f68d8505","Type":"ContainerStarted","Data":"60f8cca3477fc303c90273184515b90e107f0a9fb3e2baf961a13b48f0d8bcbc"} Oct 30 01:32:57.842474 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.842442 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" event={"ID":"0e5de0aa-5915-4679-b44f-c214cf648d4b","Type":"ContainerStarted","Data":"53b16bb8d4825f67d4ab9e5948121c9fc453567fbf9bbb08b31dc8a2139e7f22"} Oct 30 01:32:57.844293 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.844258 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" event={"ID":"9ac2a367-1394-4333-92ed-d0e3677550ad","Type":"ContainerStarted","Data":"b16d685da8c41f39fbf9b34649acc7a8905050725cc0b35de41ded038a9f372d"} Oct 30 01:32:57.844716 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.844686 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:32:57.846058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:32:57.846034 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:33:04.881797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:04.881731 5035 generic.go:356] "Generic (PLEG): container finished" podID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerID="0f311f091ef26d522ac7b4177a0e8b6286639978d8907dea903d99905edf1cf5" exitCode=0 Oct 30 01:33:04.882169 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:04.881816 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" event={"ID":"9af8d363-0df7-4e68-af45-a6b1f68d8505","Type":"ContainerDied","Data":"0f311f091ef26d522ac7b4177a0e8b6286639978d8907dea903d99905edf1cf5"} Oct 30 01:33:04.883376 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:04.883348 5035 generic.go:356] "Generic (PLEG): container finished" podID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerID="0e88ba23d5cc29723acd53e27392a949205792cd3f3fbee73603fc7556280cfc" exitCode=0 Oct 30 01:33:04.883489 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:04.883419 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" event={"ID":"0e5de0aa-5915-4679-b44f-c214cf648d4b","Type":"ContainerDied","Data":"0e88ba23d5cc29723acd53e27392a949205792cd3f3fbee73603fc7556280cfc"} Oct 30 01:33:04.885077 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:04.885056 5035 generic.go:356] "Generic (PLEG): container finished" podID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerID="00c3d202a5e1373c445d547bee653bdcb3d5bf94e76619a8be5dd75e0b07162c" exitCode=0 Oct 30 01:33:04.885216 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:04.885168 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" event={"ID":"9ac2a367-1394-4333-92ed-d0e3677550ad","Type":"ContainerDied","Data":"00c3d202a5e1373c445d547bee653bdcb3d5bf94e76619a8be5dd75e0b07162c"} Oct 30 01:33:05.891144 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:05.891112 5035 generic.go:356] "Generic (PLEG): container finished" podID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerID="2935e1104795a541690955424b5a4e781cf011ca1e0a8b9161171218fce9ac8b" exitCode=0 Oct 30 01:33:05.891496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:05.891192 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" event={"ID":"9af8d363-0df7-4e68-af45-a6b1f68d8505","Type":"ContainerDied","Data":"2935e1104795a541690955424b5a4e781cf011ca1e0a8b9161171218fce9ac8b"} Oct 30 01:33:05.893034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:05.893015 5035 generic.go:356] "Generic (PLEG): container finished" podID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerID="7345d3751b89ac7732986717cc70e0b42297ab9ab7779d34d1de1b4f449f18c7" exitCode=0 Oct 30 01:33:05.893129 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:05.893055 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" event={"ID":"9ac2a367-1394-4333-92ed-d0e3677550ad","Type":"ContainerDied","Data":"7345d3751b89ac7732986717cc70e0b42297ab9ab7779d34d1de1b4f449f18c7"} Oct 30 01:33:06.901378 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:06.901340 5035 generic.go:356] "Generic (PLEG): container finished" podID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerID="ad8691cec67bd7b6afee90d531832d327e226943c5d465893dba1199f01b6db7" exitCode=0 Oct 30 01:33:06.901836 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:06.901476 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" event={"ID":"0e5de0aa-5915-4679-b44f-c214cf648d4b","Type":"ContainerDied","Data":"ad8691cec67bd7b6afee90d531832d327e226943c5d465893dba1199f01b6db7"} Oct 30 01:33:13.942158 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:13.942123 5035 generic.go:356] "Generic (PLEG): container finished" podID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerID="0b76a39e15d156d0aee2b53b88f5a8b526589c65e539d50761097f660fea899b" exitCode=0 Oct 30 01:33:13.942631 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:13.942211 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" event={"ID":"9af8d363-0df7-4e68-af45-a6b1f68d8505","Type":"ContainerDied","Data":"0b76a39e15d156d0aee2b53b88f5a8b526589c65e539d50761097f660fea899b"} Oct 30 01:33:13.944784 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:13.944755 5035 generic.go:356] "Generic (PLEG): container finished" podID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerID="968568ac9363c3ded6db6799262d3ea33f7a18759a085e59da7b921637359273" exitCode=0 Oct 30 01:33:13.944900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:13.944810 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" event={"ID":"9ac2a367-1394-4333-92ed-d0e3677550ad","Type":"ContainerDied","Data":"968568ac9363c3ded6db6799262d3ea33f7a18759a085e59da7b921637359273"} Oct 30 01:33:14.950479 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:14.950451 5035 generic.go:356] "Generic (PLEG): container finished" podID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerID="48353a4bfa9a9cb9e4bb69053556e749bedbfa7ac64ab5b7cba123ce78a2e358" exitCode=0 Oct 30 01:33:14.950879 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:14.950521 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" event={"ID":"0e5de0aa-5915-4679-b44f-c214cf648d4b","Type":"ContainerDied","Data":"48353a4bfa9a9cb9e4bb69053556e749bedbfa7ac64ab5b7cba123ce78a2e358"} Oct 30 01:33:15.094331 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.094311 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:33:15.097258 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.097240 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:33:15.245260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.245192 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-bundle\") pod \"9ac2a367-1394-4333-92ed-d0e3677550ad\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " Oct 30 01:33:15.245260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.245227 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-bundle\") pod \"9af8d363-0df7-4e68-af45-a6b1f68d8505\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " Oct 30 01:33:15.245260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.245258 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac2a367-1394-4333-92ed-d0e3677550ad-kube-api-access-qgslx\" (UniqueName: \"kubernetes.io/projected/9ac2a367-1394-4333-92ed-d0e3677550ad-kube-api-access-qgslx\") pod \"9ac2a367-1394-4333-92ed-d0e3677550ad\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " Oct 30 01:33:15.245625 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.245289 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-util\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-util\") pod \"9ac2a367-1394-4333-92ed-d0e3677550ad\" (UID: \"9ac2a367-1394-4333-92ed-d0e3677550ad\") " Oct 30 01:33:15.245625 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.245353 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-util\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-util\") pod \"9af8d363-0df7-4e68-af45-a6b1f68d8505\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " Oct 30 01:33:15.245625 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.245381 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af8d363-0df7-4e68-af45-a6b1f68d8505-kube-api-access-vtjxl\" (UniqueName: \"kubernetes.io/projected/9af8d363-0df7-4e68-af45-a6b1f68d8505-kube-api-access-vtjxl\") pod \"9af8d363-0df7-4e68-af45-a6b1f68d8505\" (UID: \"9af8d363-0df7-4e68-af45-a6b1f68d8505\") " Oct 30 01:33:15.245784 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.245766 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-bundle" pod "9af8d363-0df7-4e68-af45-a6b1f68d8505" (UID: "9af8d363-0df7-4e68-af45-a6b1f68d8505"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:15.246031 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.246003 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-bundle" pod "9ac2a367-1394-4333-92ed-d0e3677550ad" (UID: "9ac2a367-1394-4333-92ed-d0e3677550ad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:15.247583 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.247554 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af8d363-0df7-4e68-af45-a6b1f68d8505-kube-api-access-vtjxl" pod "9af8d363-0df7-4e68-af45-a6b1f68d8505" (UID: "9af8d363-0df7-4e68-af45-a6b1f68d8505"). InnerVolumeSpecName "kube-api-access-vtjxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:15.247823 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.247802 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac2a367-1394-4333-92ed-d0e3677550ad-kube-api-access-qgslx" pod "9ac2a367-1394-4333-92ed-d0e3677550ad" (UID: "9ac2a367-1394-4333-92ed-d0e3677550ad"). InnerVolumeSpecName "kube-api-access-qgslx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:15.251212 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.251184 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-util" pod "9ac2a367-1394-4333-92ed-d0e3677550ad" (UID: "9ac2a367-1394-4333-92ed-d0e3677550ad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:15.251858 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.251832 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-util" pod "9af8d363-0df7-4e68-af45-a6b1f68d8505" (UID: "9af8d363-0df7-4e68-af45-a6b1f68d8505"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:15.346265 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.346240 5035 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:33:15.346265 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.346264 5035 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:33:15.346384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.346274 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgslx\" (UniqueName: \"kubernetes.io/projected/9ac2a367-1394-4333-92ed-d0e3677550ad-kube-api-access-qgslx\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:33:15.346384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.346284 5035 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ac2a367-1394-4333-92ed-d0e3677550ad-util\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:33:15.346384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.346292 5035 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af8d363-0df7-4e68-af45-a6b1f68d8505-util\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:33:15.346384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.346300 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtjxl\" (UniqueName: \"kubernetes.io/projected/9af8d363-0df7-4e68-af45-a6b1f68d8505-kube-api-access-vtjxl\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:33:15.957136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.957108 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" Oct 30 01:33:15.957136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.957123 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j7sx9" event={"ID":"9af8d363-0df7-4e68-af45-a6b1f68d8505","Type":"ContainerDied","Data":"60f8cca3477fc303c90273184515b90e107f0a9fb3e2baf961a13b48f0d8bcbc"} Oct 30 01:33:15.957686 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.957163 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f8cca3477fc303c90273184515b90e107f0a9fb3e2baf961a13b48f0d8bcbc" Oct 30 01:33:15.959180 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.959155 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" event={"ID":"9ac2a367-1394-4333-92ed-d0e3677550ad","Type":"ContainerDied","Data":"b16d685da8c41f39fbf9b34649acc7a8905050725cc0b35de41ded038a9f372d"} Oct 30 01:33:15.959308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.959195 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b16d685da8c41f39fbf9b34649acc7a8905050725cc0b35de41ded038a9f372d" Oct 30 01:33:15.959308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:15.959286 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9vvg" Oct 30 01:33:16.083684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.083660 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:33:16.153580 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.153555 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-util\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-util\") pod \"0e5de0aa-5915-4679-b44f-c214cf648d4b\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " Oct 30 01:33:16.153731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.153598 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-bundle\") pod \"0e5de0aa-5915-4679-b44f-c214cf648d4b\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " Oct 30 01:33:16.153731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.153669 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5de0aa-5915-4679-b44f-c214cf648d4b-kube-api-access-xd8gl\" (UniqueName: \"kubernetes.io/projected/0e5de0aa-5915-4679-b44f-c214cf648d4b-kube-api-access-xd8gl\") pod \"0e5de0aa-5915-4679-b44f-c214cf648d4b\" (UID: \"0e5de0aa-5915-4679-b44f-c214cf648d4b\") " Oct 30 01:33:16.154147 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.154125 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-bundle" pod "0e5de0aa-5915-4679-b44f-c214cf648d4b" (UID: "0e5de0aa-5915-4679-b44f-c214cf648d4b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:16.155720 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.155696 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5de0aa-5915-4679-b44f-c214cf648d4b-kube-api-access-xd8gl" pod "0e5de0aa-5915-4679-b44f-c214cf648d4b" (UID: "0e5de0aa-5915-4679-b44f-c214cf648d4b"). InnerVolumeSpecName "kube-api-access-xd8gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:16.159186 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.159160 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-util" pod "0e5de0aa-5915-4679-b44f-c214cf648d4b" (UID: "0e5de0aa-5915-4679-b44f-c214cf648d4b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:16.254765 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.254702 5035 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-util\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:33:16.254765 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.254727 5035 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5de0aa-5915-4679-b44f-c214cf648d4b-bundle\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:33:16.254765 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.254740 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd8gl\" (UniqueName: \"kubernetes.io/projected/0e5de0aa-5915-4679-b44f-c214cf648d4b-kube-api-access-xd8gl\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:33:16.965201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.965167 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" event={"ID":"0e5de0aa-5915-4679-b44f-c214cf648d4b","Type":"ContainerDied","Data":"53b16bb8d4825f67d4ab9e5948121c9fc453567fbf9bbb08b31dc8a2139e7f22"} Oct 30 01:33:16.965588 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.965220 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53b16bb8d4825f67d4ab9e5948121c9fc453567fbf9bbb08b31dc8a2139e7f22" Oct 30 01:33:16.965588 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:16.965174 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ss68d" Oct 30 01:33:17.625675 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:33:17.625637 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:33:29.952492 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952460 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d"] Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952782 5035 cpu_manager.go:414] "containerMap: removing container" podUID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerName="pull" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952792 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerName="pull" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952805 5035 cpu_manager.go:414] "containerMap: removing container" podUID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerName="extract" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952810 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerName="extract" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952819 5035 cpu_manager.go:414] "containerMap: removing container" podUID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerName="extract" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952824 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerName="extract" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952832 5035 cpu_manager.go:414] "containerMap: removing container" podUID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerName="util" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952836 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerName="util" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952845 5035 cpu_manager.go:414] "containerMap: removing container" podUID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerName="pull" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952849 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerName="pull" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952858 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerName="util" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952863 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerName="util" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952870 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerName="extract" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952874 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerName="extract" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952884 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerName="pull" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952888 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerName="pull" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952907 5035 cpu_manager.go:414] "containerMap: removing container" podUID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerName="util" Oct 30 01:33:29.952929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952912 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerName="util" Oct 30 01:33:29.953505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952964 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac2a367-1394-4333-92ed-d0e3677550ad" containerName="extract" Oct 30 01:33:29.953505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952972 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af8d363-0df7-4e68-af45-a6b1f68d8505" containerName="extract" Oct 30 01:33:29.953505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.952979 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5de0aa-5915-4679-b44f-c214cf648d4b" containerName="extract" Oct 30 01:33:29.955834 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.955816 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:29.976064 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.976036 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d"] Oct 30 01:33:29.976277 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:29.976260 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-79ppx\"" Oct 30 01:33:30.042453 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.042427 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78213f32-8d95-4766-b1e4-564c85ab2eae-webhook-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.042617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.042464 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78213f32-8d95-4766-b1e4-564c85ab2eae-metrics-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.042617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.042483 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78213f32-8d95-4766-b1e4-564c85ab2eae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.042617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.042564 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rcvz\" (UniqueName: \"kubernetes.io/projected/78213f32-8d95-4766-b1e4-564c85ab2eae-kube-api-access-2rcvz\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.042730 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.042612 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78213f32-8d95-4766-b1e4-564c85ab2eae-tmp\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.042730 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.042641 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78213f32-8d95-4766-b1e4-564c85ab2eae-manifests\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.045620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.045603 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 01:33:30.045817 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.045802 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 01:33:30.046138 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.046125 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 01:33:30.046183 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.046130 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 01:33:30.046411 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.046398 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 01:33:30.143611 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.143586 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78213f32-8d95-4766-b1e4-564c85ab2eae-metrics-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.143731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.143613 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78213f32-8d95-4766-b1e4-564c85ab2eae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.143731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.143636 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcvz\" (UniqueName: \"kubernetes.io/projected/78213f32-8d95-4766-b1e4-564c85ab2eae-kube-api-access-2rcvz\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.143731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.143653 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78213f32-8d95-4766-b1e4-564c85ab2eae-tmp\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.143731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.143670 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78213f32-8d95-4766-b1e4-564c85ab2eae-manifests\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.143931 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.143740 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78213f32-8d95-4766-b1e4-564c85ab2eae-webhook-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.144196 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.144165 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78213f32-8d95-4766-b1e4-564c85ab2eae-manifests\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.146103 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.146072 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78213f32-8d95-4766-b1e4-564c85ab2eae-tmp\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.146193 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.146114 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78213f32-8d95-4766-b1e4-564c85ab2eae-metrics-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.146193 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.146137 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78213f32-8d95-4766-b1e4-564c85ab2eae-webhook-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.146268 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.146240 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78213f32-8d95-4766-b1e4-564c85ab2eae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.167870 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.167846 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcvz\" (UniqueName: \"kubernetes.io/projected/78213f32-8d95-4766-b1e4-564c85ab2eae-kube-api-access-2rcvz\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-zns6d\" (UID: \"78213f32-8d95-4766-b1e4-564c85ab2eae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.270183 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.270115 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:30.627173 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:30.627047 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d"] Oct 30 01:33:30.629726 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:33:30.629697 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78213f32_8d95_4766_b1e4_564c85ab2eae.slice/crio-9028053ccce5de98a58a03a3b351635a8db9cfa5f933e1b6298d631bcd411983 WatchSource:0}: Error finding container 9028053ccce5de98a58a03a3b351635a8db9cfa5f933e1b6298d631bcd411983: Status 404 returned error can't find the container with id 9028053ccce5de98a58a03a3b351635a8db9cfa5f933e1b6298d631bcd411983 Oct 30 01:33:31.020969 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:31.020931 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" event={"ID":"78213f32-8d95-4766-b1e4-564c85ab2eae","Type":"ContainerStarted","Data":"9028053ccce5de98a58a03a3b351635a8db9cfa5f933e1b6298d631bcd411983"} Oct 30 01:33:34.040307 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:34.040253 5035 generic.go:356] "Generic (PLEG): container finished" podID="78213f32-8d95-4766-b1e4-564c85ab2eae" containerID="263c692f8da4551419abeecd67a2c99304e0445b5e38ff6eddc4b3fd9aeb8b25" exitCode=0 Oct 30 01:33:34.040789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:34.040317 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" event={"ID":"78213f32-8d95-4766-b1e4-564c85ab2eae","Type":"ContainerDied","Data":"263c692f8da4551419abeecd67a2c99304e0445b5e38ff6eddc4b3fd9aeb8b25"} Oct 30 01:33:35.047266 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:35.047229 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" event={"ID":"78213f32-8d95-4766-b1e4-564c85ab2eae","Type":"ContainerStarted","Data":"a9ccb3d9d4316db1826f757f21f823a9ed9f1bed16fa047c6696e27e87bc918b"} Oct 30 01:33:35.047676 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:35.047303 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:35.149731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:35.149679 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" podStartSLOduration=2.5520879709999997 podStartE2EDuration="6.149663687s" totalImagesPullingTime="3.141439428s" totalInitContainerRuntime="456.136288ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:35.136047761 +0000 UTC m=+293.638979981" watchObservedRunningTime="2026-10-30 01:33:35.149663687 +0000 UTC m=+293.652595908" Oct 30 01:33:41.664179 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:41.664154 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:33:41.664659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:41.664252 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:33:44.301117 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.301063 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj"] Oct 30 01:33:44.307819 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.307790 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" Oct 30 01:33:44.328658 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.328633 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj"] Oct 30 01:33:44.339605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.339581 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 01:33:44.339691 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.339619 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 01:33:44.342920 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.342904 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-h22hr\"" Oct 30 01:33:44.358751 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.358730 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2hds\" (UniqueName: \"kubernetes.io/projected/adaf904e-ceb8-48d6-816f-3316258fe072-kube-api-access-f2hds\") pod \"limitador-operator-controller-manager-5778bb4449-57dvj\" (UID: \"adaf904e-ceb8-48d6-816f-3316258fe072\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" Oct 30 01:33:44.459907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.459880 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hds\" (UniqueName: \"kubernetes.io/projected/adaf904e-ceb8-48d6-816f-3316258fe072-kube-api-access-f2hds\") pod \"limitador-operator-controller-manager-5778bb4449-57dvj\" (UID: \"adaf904e-ceb8-48d6-816f-3316258fe072\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" Oct 30 01:33:44.484305 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.484282 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hds\" (UniqueName: \"kubernetes.io/projected/adaf904e-ceb8-48d6-816f-3316258fe072-kube-api-access-f2hds\") pod \"limitador-operator-controller-manager-5778bb4449-57dvj\" (UID: \"adaf904e-ceb8-48d6-816f-3316258fe072\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" Oct 30 01:33:44.620167 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.620104 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" Oct 30 01:33:44.916133 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.916106 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj"] Oct 30 01:33:44.918657 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:33:44.918613 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadaf904e_ceb8_48d6_816f_3316258fe072.slice/crio-231fd2b185bff8403a453d2602312d64a2285d3495106e078956fc43aa2f7f78 WatchSource:0}: Error finding container 231fd2b185bff8403a453d2602312d64a2285d3495106e078956fc43aa2f7f78: Status 404 returned error can't find the container with id 231fd2b185bff8403a453d2602312d64a2285d3495106e078956fc43aa2f7f78 Oct 30 01:33:44.921010 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:44.920993 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:33:45.090934 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:45.090904 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" event={"ID":"adaf904e-ceb8-48d6-816f-3316258fe072","Type":"ContainerStarted","Data":"231fd2b185bff8403a453d2602312d64a2285d3495106e078956fc43aa2f7f78"} Oct 30 01:33:46.054752 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:46.054725 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-zns6d" Oct 30 01:33:47.626429 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:33:47.626394 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:33:52.123012 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:52.122925 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" event={"ID":"adaf904e-ceb8-48d6-816f-3316258fe072","Type":"ContainerStarted","Data":"aeaba22284347794ae96c8ac7ba02fa295096c63c160a61fd73ec7c8334f8242"} Oct 30 01:33:52.123367 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:52.123019 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" Oct 30 01:33:52.190499 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:52.190456 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" podStartSLOduration=1.403183149 podStartE2EDuration="8.190442233s" totalImagesPullingTime="6.787259084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:52.188923541 +0000 UTC m=+310.691855763" watchObservedRunningTime="2026-10-30 01:33:52.190442233 +0000 UTC m=+310.693374449" Oct 30 01:33:59.450651 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.450572 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j"] Oct 30 01:33:59.457755 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.457728 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" Oct 30 01:33:59.472884 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.472857 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j"] Oct 30 01:33:59.483912 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.483884 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-75zs6\"" Oct 30 01:33:59.483912 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.483900 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 01:33:59.484072 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.483953 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 01:33:59.577460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.577422 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4tv8\" (UniqueName: \"kubernetes.io/projected/1ae7b5f0-a85b-4145-bd61-8b70bbb75323-kube-api-access-f4tv8\") pod \"cert-manager-cainjector-5c57bbd4fc-f8g6j\" (UID: \"1ae7b5f0-a85b-4145-bd61-8b70bbb75323\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" Oct 30 01:33:59.577599 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.577468 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ae7b5f0-a85b-4145-bd61-8b70bbb75323-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-f8g6j\" (UID: \"1ae7b5f0-a85b-4145-bd61-8b70bbb75323\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" Oct 30 01:33:59.678389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.678348 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tv8\" (UniqueName: \"kubernetes.io/projected/1ae7b5f0-a85b-4145-bd61-8b70bbb75323-kube-api-access-f4tv8\") pod \"cert-manager-cainjector-5c57bbd4fc-f8g6j\" (UID: \"1ae7b5f0-a85b-4145-bd61-8b70bbb75323\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" Oct 30 01:33:59.678389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.678393 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ae7b5f0-a85b-4145-bd61-8b70bbb75323-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-f8g6j\" (UID: \"1ae7b5f0-a85b-4145-bd61-8b70bbb75323\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" Oct 30 01:33:59.702779 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.702718 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ae7b5f0-a85b-4145-bd61-8b70bbb75323-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-f8g6j\" (UID: \"1ae7b5f0-a85b-4145-bd61-8b70bbb75323\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" Oct 30 01:33:59.702878 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.702807 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tv8\" (UniqueName: \"kubernetes.io/projected/1ae7b5f0-a85b-4145-bd61-8b70bbb75323-kube-api-access-f4tv8\") pod \"cert-manager-cainjector-5c57bbd4fc-f8g6j\" (UID: \"1ae7b5f0-a85b-4145-bd61-8b70bbb75323\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" Oct 30 01:33:59.780198 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:33:59.780161 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" Oct 30 01:34:00.075358 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:00.075194 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j"] Oct 30 01:34:00.077990 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:34:00.077954 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ae7b5f0_a85b_4145_bd61_8b70bbb75323.slice/crio-e6e01825d4f58d54ab8af98d29d7ef1b5518d11b9e51f39236defb5e1b65f2bf WatchSource:0}: Error finding container e6e01825d4f58d54ab8af98d29d7ef1b5518d11b9e51f39236defb5e1b65f2bf: Status 404 returned error can't find the container with id e6e01825d4f58d54ab8af98d29d7ef1b5518d11b9e51f39236defb5e1b65f2bf Oct 30 01:34:00.159308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:00.159277 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" event={"ID":"1ae7b5f0-a85b-4145-bd61-8b70bbb75323","Type":"ContainerStarted","Data":"e6e01825d4f58d54ab8af98d29d7ef1b5518d11b9e51f39236defb5e1b65f2bf"} Oct 30 01:34:01.275783 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.275748 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-45k2d"] Oct 30 01:34:01.278776 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.278755 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" Oct 30 01:34:01.302071 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.302030 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nckqx\"" Oct 30 01:34:01.315065 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.314996 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-45k2d"] Oct 30 01:34:01.394392 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.394358 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mpnm\" (UniqueName: \"kubernetes.io/projected/e33eacd4-92b9-47b7-b441-7f0b47bd6071-kube-api-access-2mpnm\") pod \"authorino-operator-7b7bb4bc77-45k2d\" (UID: \"e33eacd4-92b9-47b7-b441-7f0b47bd6071\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" Oct 30 01:34:01.496110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.496070 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mpnm\" (UniqueName: \"kubernetes.io/projected/e33eacd4-92b9-47b7-b441-7f0b47bd6071-kube-api-access-2mpnm\") pod \"authorino-operator-7b7bb4bc77-45k2d\" (UID: \"e33eacd4-92b9-47b7-b441-7f0b47bd6071\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" Oct 30 01:34:01.528272 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.528165 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mpnm\" (UniqueName: \"kubernetes.io/projected/e33eacd4-92b9-47b7-b441-7f0b47bd6071-kube-api-access-2mpnm\") pod \"authorino-operator-7b7bb4bc77-45k2d\" (UID: \"e33eacd4-92b9-47b7-b441-7f0b47bd6071\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" Oct 30 01:34:01.593890 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.593858 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" Oct 30 01:34:01.745304 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.743449 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv"] Oct 30 01:34:01.746954 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.746928 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.765696 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.765669 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv"] Oct 30 01:34:01.770449 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.770417 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5v54n\"" Oct 30 01:34:01.799481 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.799301 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.800541 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.799705 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.800541 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.799772 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.800541 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.799819 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.800541 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.799879 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.800541 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.799923 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.800541 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.799963 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmwz\" (UniqueName: \"kubernetes.io/projected/a3315f88-afe8-4cf3-9f2b-16567421aae8-kube-api-access-xkmwz\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.800541 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.799996 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.800541 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.800049 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3315f88-afe8-4cf3-9f2b-16567421aae8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.901484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.901451 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.901671 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.901545 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.901671 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.901618 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.901848 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.901744 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.902236 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.902201 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.902346 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.902332 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.903869 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.903843 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.905560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.903902 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.905560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.903941 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmwz\" (UniqueName: \"kubernetes.io/projected/a3315f88-afe8-4cf3-9f2b-16567421aae8-kube-api-access-xkmwz\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.905560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.903973 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.905560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.904016 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3315f88-afe8-4cf3-9f2b-16567421aae8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.905560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.904341 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.905560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.904652 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.905560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.904938 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.905560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.905255 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3315f88-afe8-4cf3-9f2b-16567421aae8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.906358 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.906332 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.921693 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.921665 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-45k2d"] Oct 30 01:34:01.927739 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.927277 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3315f88-afe8-4cf3-9f2b-16567421aae8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:01.937086 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:01.937041 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmwz\" (UniqueName: \"kubernetes.io/projected/a3315f88-afe8-4cf3-9f2b-16567421aae8-kube-api-access-xkmwz\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv\" (UID: \"a3315f88-afe8-4cf3-9f2b-16567421aae8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:02.063747 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.063664 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:02.172866 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.172795 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" event={"ID":"e33eacd4-92b9-47b7-b441-7f0b47bd6071","Type":"ContainerStarted","Data":"2e0fd094f74f788070742cc7f7f5a5972be8d069feb769dd6009e5758edb1a6e"} Oct 30 01:34:02.400135 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.400098 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv"] Oct 30 01:34:02.402131 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:34:02.402100 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3315f88_afe8_4cf3_9f2b_16567421aae8.slice/crio-a0555717022d03b19904dfdfb422f013a58a74bffcea27b3dbeb4eacee61b4e0 WatchSource:0}: Error finding container a0555717022d03b19904dfdfb422f013a58a74bffcea27b3dbeb4eacee61b4e0: Status 404 returned error can't find the container with id a0555717022d03b19904dfdfb422f013a58a74bffcea27b3dbeb4eacee61b4e0 Oct 30 01:34:02.404594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.404562 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:02.404689 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.404638 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:02.404751 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.404687 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:02.674172 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.672465 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf"] Oct 30 01:34:02.677819 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.677800 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:02.704093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.698294 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf"] Oct 30 01:34:02.704093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.699075 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 01:34:02.704093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.699191 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w4cxk\"" Oct 30 01:34:02.704093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.699236 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 01:34:02.814583 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.814548 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e497615-20c6-4568-a41b-f8220a1f2e6c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8fhqf\" (UID: \"6e497615-20c6-4568-a41b-f8220a1f2e6c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:02.814788 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.814618 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kb7\" (UniqueName: \"kubernetes.io/projected/6e497615-20c6-4568-a41b-f8220a1f2e6c-kube-api-access-b9kb7\") pod \"kuadrant-console-plugin-66c4fb575d-8fhqf\" (UID: \"6e497615-20c6-4568-a41b-f8220a1f2e6c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:02.814788 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.814658 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e497615-20c6-4568-a41b-f8220a1f2e6c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8fhqf\" (UID: \"6e497615-20c6-4568-a41b-f8220a1f2e6c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:02.915762 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.915708 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e497615-20c6-4568-a41b-f8220a1f2e6c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8fhqf\" (UID: \"6e497615-20c6-4568-a41b-f8220a1f2e6c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:02.916006 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.915789 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kb7\" (UniqueName: \"kubernetes.io/projected/6e497615-20c6-4568-a41b-f8220a1f2e6c-kube-api-access-b9kb7\") pod \"kuadrant-console-plugin-66c4fb575d-8fhqf\" (UID: \"6e497615-20c6-4568-a41b-f8220a1f2e6c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:02.916006 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.915839 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e497615-20c6-4568-a41b-f8220a1f2e6c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8fhqf\" (UID: \"6e497615-20c6-4568-a41b-f8220a1f2e6c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:02.917474 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.917440 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e497615-20c6-4568-a41b-f8220a1f2e6c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8fhqf\" (UID: \"6e497615-20c6-4568-a41b-f8220a1f2e6c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:02.920282 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.920234 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e497615-20c6-4568-a41b-f8220a1f2e6c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8fhqf\" (UID: \"6e497615-20c6-4568-a41b-f8220a1f2e6c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:02.941854 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:02.941790 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kb7\" (UniqueName: \"kubernetes.io/projected/6e497615-20c6-4568-a41b-f8220a1f2e6c-kube-api-access-b9kb7\") pod \"kuadrant-console-plugin-66c4fb575d-8fhqf\" (UID: \"6e497615-20c6-4568-a41b-f8220a1f2e6c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:03.011397 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:03.011361 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" Oct 30 01:34:03.129628 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:03.129593 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-57dvj" Oct 30 01:34:03.188797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:03.188721 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" event={"ID":"a3315f88-afe8-4cf3-9f2b-16567421aae8","Type":"ContainerStarted","Data":"63167eddb597e17b2a75dd601ea4ce95ffca3795fd8f3d481304f705b1cc4c91"} Oct 30 01:34:03.189133 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:03.189083 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" event={"ID":"a3315f88-afe8-4cf3-9f2b-16567421aae8","Type":"ContainerStarted","Data":"a0555717022d03b19904dfdfb422f013a58a74bffcea27b3dbeb4eacee61b4e0"} Oct 30 01:34:03.257143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:03.257081 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" podStartSLOduration=2.257066165 podStartE2EDuration="2.257066165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:03.255297428 +0000 UTC m=+321.758229649" watchObservedRunningTime="2026-10-30 01:34:03.257066165 +0000 UTC m=+321.759998385" Oct 30 01:34:03.318421 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:03.318365 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf"] Oct 30 01:34:03.322778 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:34:03.322736 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e497615_20c6_4568_a41b_f8220a1f2e6c.slice/crio-e3468afe5845793330669f3d79b6a3eae5de5e0487202834b6c01b81bda7c810 WatchSource:0}: Error finding container e3468afe5845793330669f3d79b6a3eae5de5e0487202834b6c01b81bda7c810: Status 404 returned error can't find the container with id e3468afe5845793330669f3d79b6a3eae5de5e0487202834b6c01b81bda7c810 Oct 30 01:34:04.064466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.064429 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:04.069706 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.069676 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:04.199452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.199406 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" event={"ID":"6e497615-20c6-4568-a41b-f8220a1f2e6c","Type":"ContainerStarted","Data":"e3468afe5845793330669f3d79b6a3eae5de5e0487202834b6c01b81bda7c810"} Oct 30 01:34:04.202587 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.202558 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" event={"ID":"e33eacd4-92b9-47b7-b441-7f0b47bd6071","Type":"ContainerStarted","Data":"40a10d54ff64d4fb0c6fdbb5a934f3e7ed7f37e8bcd1278f7deb037c816b863d"} Oct 30 01:34:04.202804 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.202753 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" Oct 30 01:34:04.205133 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.205108 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" event={"ID":"1ae7b5f0-a85b-4145-bd61-8b70bbb75323","Type":"ContainerStarted","Data":"8d5659c272b1a047a01eb380ab35b606b030973e920fd18b6bdf4c79ff14cf83"} Oct 30 01:34:04.205243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.205222 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:04.206472 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.206428 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv" Oct 30 01:34:04.253617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.253558 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" podStartSLOduration=1.496773895 podStartE2EDuration="3.253427669s" totalImagesPullingTime="1.756653774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:04.252408439 +0000 UTC m=+322.755340658" watchObservedRunningTime="2026-10-30 01:34:04.253427669 +0000 UTC m=+322.756359889" Oct 30 01:34:04.305426 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:04.305355 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f8g6j" podStartSLOduration=1.598745723 podStartE2EDuration="5.305337069s" totalImagesPullingTime="3.706591346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:04.302006829 +0000 UTC m=+322.804939050" watchObservedRunningTime="2026-10-30 01:34:04.305337069 +0000 UTC m=+322.808269291" Oct 30 01:34:10.238079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:10.238040 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" event={"ID":"6e497615-20c6-4568-a41b-f8220a1f2e6c","Type":"ContainerStarted","Data":"61cd8d232e8d90d171f51411020a04afbe11231b84bbb60bf8eb09bbb46f44af"} Oct 30 01:34:10.280751 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:10.280704 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8fhqf" podStartSLOduration=2.100955208 podStartE2EDuration="8.280683209s" totalImagesPullingTime="6.179728001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:10.279851273 +0000 UTC m=+328.782783493" watchObservedRunningTime="2026-10-30 01:34:10.280683209 +0000 UTC m=+328.783615429" Oct 30 01:34:15.213048 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:15.213009 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-45k2d" Oct 30 01:34:17.625691 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:34:17.625654 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:34:30.054022 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.053990 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6"] Oct 30 01:34:30.062112 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.062090 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.078448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.078423 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6"] Oct 30 01:34:30.090035 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.090006 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 01:34:30.090907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.090888 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 01:34:30.091014 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.090930 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-cgv94\"" Oct 30 01:34:30.091014 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.090965 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:34:30.091261 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.091243 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 01:34:30.091298 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.091250 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 01:34:30.247958 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.247931 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e18c660b-841b-49db-aaf6-e84bcef4c34b-cert\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.248105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.247975 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntg9\" (UniqueName: \"kubernetes.io/projected/e18c660b-841b-49db-aaf6-e84bcef4c34b-kube-api-access-hntg9\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.248105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.248006 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18c660b-841b-49db-aaf6-e84bcef4c34b-metrics-certs\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.248180 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.248135 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e18c660b-841b-49db-aaf6-e84bcef4c34b-manager-config\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.349582 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.349491 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e18c660b-841b-49db-aaf6-e84bcef4c34b-manager-config\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.349582 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.349540 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e18c660b-841b-49db-aaf6-e84bcef4c34b-cert\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.349791 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.349611 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntg9\" (UniqueName: \"kubernetes.io/projected/e18c660b-841b-49db-aaf6-e84bcef4c34b-kube-api-access-hntg9\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.349791 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.349675 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18c660b-841b-49db-aaf6-e84bcef4c34b-metrics-certs\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.350175 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.350151 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e18c660b-841b-49db-aaf6-e84bcef4c34b-manager-config\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.352076 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.352051 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18c660b-841b-49db-aaf6-e84bcef4c34b-metrics-certs\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.352166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.352122 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e18c660b-841b-49db-aaf6-e84bcef4c34b-cert\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.374314 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.374285 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntg9\" (UniqueName: \"kubernetes.io/projected/e18c660b-841b-49db-aaf6-e84bcef4c34b-kube-api-access-hntg9\") pod \"jobset-controller-manager-5c89ff7c87-fpdp6\" (UID: \"e18c660b-841b-49db-aaf6-e84bcef4c34b\") " pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.374466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.374440 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:30.706918 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:30.706884 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6"] Oct 30 01:34:30.709695 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:34:30.709662 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18c660b_841b_49db_aaf6_e84bcef4c34b.slice/crio-c31f19d029fc8da41cae08b2a42525e682d6ab2322660429f275e2318a2584ab WatchSource:0}: Error finding container c31f19d029fc8da41cae08b2a42525e682d6ab2322660429f275e2318a2584ab: Status 404 returned error can't find the container with id c31f19d029fc8da41cae08b2a42525e682d6ab2322660429f275e2318a2584ab Oct 30 01:34:31.336454 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:31.336410 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" event={"ID":"e18c660b-841b-49db-aaf6-e84bcef4c34b","Type":"ContainerStarted","Data":"c31f19d029fc8da41cae08b2a42525e682d6ab2322660429f275e2318a2584ab"} Oct 30 01:34:33.348035 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:33.347999 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" event={"ID":"e18c660b-841b-49db-aaf6-e84bcef4c34b","Type":"ContainerStarted","Data":"3a4bf66a823a015c1932ee372202b8166940bf7ccc2330d3f85a53a78973b2e9"} Oct 30 01:34:33.348415 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:33.348070 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:33.392692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:33.392637 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" podStartSLOduration=0.875522187 podStartE2EDuration="3.392622627s" totalImagesPullingTime="2.51710044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:33.391655961 +0000 UTC m=+351.894588181" watchObservedRunningTime="2026-10-30 01:34:33.392622627 +0000 UTC m=+351.895554877" Oct 30 01:34:44.358209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:34:44.358176 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5c89ff7c87-fpdp6" Oct 30 01:34:47.626362 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:34:47.626327 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:35:17.626333 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:35:17.626293 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:35:47.626618 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:35:47.626578 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:36:17.626280 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:36:17.626236 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:36:47.626449 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:36:47.626413 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:37:17.625966 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:37:17.625887 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:37:47.626465 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:37:47.626424 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:38:17.625876 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:38:17.625841 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:38:41.698750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:38:41.698670 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:38:41.700254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:38:41.700231 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:38:47.627291 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:38:47.627255 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:39:17.626729 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:39:17.626689 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:39:47.627333 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:39:47.627295 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:40:17.627126 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:40:17.627038 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:40:47.626050 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:40:47.626012 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:41:17.626618 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:41:17.626581 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:41:47.626462 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:41:47.626380 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:42:17.626962 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:42:17.626914 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:42:47.626616 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:42:47.626576 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:43:17.626056 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:43:17.625958 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:43:41.737045 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:43:41.737014 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:43:41.739635 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:43:41.739609 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:43:47.625909 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:43:47.625860 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:44:17.625855 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:44:17.625818 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:44:47.625853 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:44:47.625815 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:45:17.627048 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:45:17.627013 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:45:47.626905 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:45:47.626860 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:46:02.537365 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:02.537325 5035 generic.go:356] "Generic (PLEG): container finished" podID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerID="a466a4de1b11b6eccf402360c9dc56a62dbc58114254ae93d9987a197c4627f3" exitCode=0 Oct 30 01:46:02.537835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:02.537400 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" event={"ID":"b359b740-709a-4fd9-ade6-4bf08d7b827b","Type":"ContainerDied","Data":"a466a4de1b11b6eccf402360c9dc56a62dbc58114254ae93d9987a197c4627f3"} Oct 30 01:46:02.537835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:02.537776 5035 scope.go:122] "RemoveContainer" containerID="a466a4de1b11b6eccf402360c9dc56a62dbc58114254ae93d9987a197c4627f3" Oct 30 01:46:02.538797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:02.538766 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:46:02.538909 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:02.538847 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:46:02.538909 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:02.538896 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:46:03.544603 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:03.544569 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" event={"ID":"b359b740-709a-4fd9-ade6-4bf08d7b827b","Type":"ContainerStarted","Data":"ab8ee71dd0097d79c0cbd6852124d7093c3b041546501018e4d122edf1a0e853"} Oct 30 01:46:04.240748 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:04.240716 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:46:04.240926 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:04.240861 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8pw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:46:04.240979 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:04.240916 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:46:05.241301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:05.241265 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8pw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:46:05.241747 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:05.241318 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:46:06.240459 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:06.240423 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8pw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:46:06.240659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:06.240475 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:46:07.241167 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:07.241135 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8pw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:46:07.241692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:07.241203 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:46:08.241260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:08.241223 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8pw9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:46:08.241684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:08.241301 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:46:09.244881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:09.244853 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:46:09.245302 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:09.245111 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:46:09.245785 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:46:09.245766 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 01:46:17.625843 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:46:17.625804 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:46:47.626742 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:46:47.626703 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:47:17.043137 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.043103 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw"] Oct 30 01:47:17.046490 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.046473 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.080245 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.080209 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9bctt\"" Oct 30 01:47:17.080407 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.080310 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 01:47:17.082025 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.082003 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw"] Oct 30 01:47:17.104495 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.104466 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 01:47:17.104495 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.104493 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 01:47:17.122002 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.121966 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.122173 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.122087 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l66rl\" (UniqueName: \"kubernetes.io/projected/828bd886-3208-4152-b4bb-551b10a3a254-kube-api-access-l66rl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.122173 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.122131 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.122272 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.122180 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828bd886-3208-4152-b4bb-551b10a3a254-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.122338 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.122319 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.122393 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.122363 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.223665 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.223619 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.223886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.223672 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.223886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.223700 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.223886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.223749 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l66rl\" (UniqueName: \"kubernetes.io/projected/828bd886-3208-4152-b4bb-551b10a3a254-kube-api-access-l66rl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.223886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.223781 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.223886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.223833 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828bd886-3208-4152-b4bb-551b10a3a254-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.224207 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.224178 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.224276 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.224211 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.224276 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.224229 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.226097 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.226069 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828bd886-3208-4152-b4bb-551b10a3a254-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.226245 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.226229 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828bd886-3208-4152-b4bb-551b10a3a254-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.253292 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.253256 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l66rl\" (UniqueName: \"kubernetes.io/projected/828bd886-3208-4152-b4bb-551b10a3a254-kube-api-access-l66rl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4dsxw\" (UID: \"828bd886-3208-4152-b4bb-551b10a3a254\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.362009 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.361899 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:17.628495 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:47:17.628413 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:47:17.659271 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.659242 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw"] Oct 30 01:47:17.661692 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:47:17.661666 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828bd886_3208_4152_b4bb_551b10a3a254.slice/crio-75d9e1f89bd03ff58e817bcc6ad75cc3c1632e197a0dd3bf8670d8ea3ae2ff48 WatchSource:0}: Error finding container 75d9e1f89bd03ff58e817bcc6ad75cc3c1632e197a0dd3bf8670d8ea3ae2ff48: Status 404 returned error can't find the container with id 75d9e1f89bd03ff58e817bcc6ad75cc3c1632e197a0dd3bf8670d8ea3ae2ff48 Oct 30 01:47:17.663009 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.662989 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:47:17.885806 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:17.885719 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" event={"ID":"828bd886-3208-4152-b4bb-551b10a3a254","Type":"ContainerStarted","Data":"75d9e1f89bd03ff58e817bcc6ad75cc3c1632e197a0dd3bf8670d8ea3ae2ff48"} Oct 30 01:47:18.597927 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.596048 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns"] Oct 30 01:47:18.600462 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.600430 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.626345 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.626315 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns"] Oct 30 01:47:18.633659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.633607 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 01:47:18.740623 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.740582 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnq6l\" (UniqueName: \"kubernetes.io/projected/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-kube-api-access-lnq6l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.740939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.740669 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.740939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.740735 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.740939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.740763 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.740939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.740801 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.740939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.740877 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.841317 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.841287 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnq6l\" (UniqueName: \"kubernetes.io/projected/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-kube-api-access-lnq6l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.841460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.841338 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.841460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.841384 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.841460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.841403 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.841460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.841421 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.841460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.841439 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.841811 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.841787 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.841910 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.841884 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.841964 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.841906 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.843698 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.843680 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.843933 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.843915 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.865916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.865838 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnq6l\" (UniqueName: \"kubernetes.io/projected/ef8fcd62-a30c-463d-9ffc-a516d8eb7e73-kube-api-access-lnq6l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-v5zns\" (UID: \"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.893678 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.893635 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" event={"ID":"828bd886-3208-4152-b4bb-551b10a3a254","Type":"ContainerStarted","Data":"2980be0c37031e5115256e06dc7e9333d5492348e6ee9a90a89d3efcb814b3f7"} Oct 30 01:47:18.920613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.920572 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:18.939206 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:18.939159 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" podStartSLOduration=0.84261295 podStartE2EDuration="1.939145106s" totalImagesPullingTime="1.096532156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:18.938209896 +0000 UTC m=+1117.441142132" watchObservedRunningTime="2026-10-30 01:47:18.939145106 +0000 UTC m=+1117.442077325" Oct 30 01:47:19.228197 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:19.228154 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns"] Oct 30 01:47:19.231373 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:47:19.231344 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8fcd62_a30c_463d_9ffc_a516d8eb7e73.slice/crio-1d716457ed97c61e1af80aad6413c8062e85788e08192fd6772850487aca7c72 WatchSource:0}: Error finding container 1d716457ed97c61e1af80aad6413c8062e85788e08192fd6772850487aca7c72: Status 404 returned error can't find the container with id 1d716457ed97c61e1af80aad6413c8062e85788e08192fd6772850487aca7c72 Oct 30 01:47:19.362357 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:19.362319 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:19.900285 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:19.900185 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" event={"ID":"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73","Type":"ContainerStarted","Data":"ba21620080ee94cd274f13ceecd6d021d74238e1e51acc820399740f21c22869"} Oct 30 01:47:19.900285 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:19.900239 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" event={"ID":"ef8fcd62-a30c-463d-9ffc-a516d8eb7e73","Type":"ContainerStarted","Data":"1d716457ed97c61e1af80aad6413c8062e85788e08192fd6772850487aca7c72"} Oct 30 01:47:19.921019 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:19.920991 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:19.945087 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:19.945037 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" podStartSLOduration=1.765183166 podStartE2EDuration="1.94502236s" totalImagesPullingTime="179.839194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:19.944690981 +0000 UTC m=+1118.447623201" watchObservedRunningTime="2026-10-30 01:47:19.94502236 +0000 UTC m=+1118.447954618" Oct 30 01:47:22.702749 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.702711 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dw8g7"] Oct 30 01:47:22.706479 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.706457 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.727588 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.727547 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dw8g7"] Oct 30 01:47:22.735255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.735227 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 01:47:22.779165 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.779123 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.779165 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.779168 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.779389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.779327 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.779389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.779345 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmnx\" (UniqueName: \"kubernetes.io/projected/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-kube-api-access-stmnx\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.779389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.779377 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-home\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.779583 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.779398 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.880225 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.880142 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-home\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.880225 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.880182 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.880225 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.880210 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.880518 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.880228 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.880518 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.880419 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.880518 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.880446 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmnx\" (UniqueName: \"kubernetes.io/projected/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-kube-api-access-stmnx\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.880717 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.880689 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.880771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.880695 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-home\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.880817 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.880778 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.882654 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.882630 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.882919 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.882901 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:22.905696 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:22.905652 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmnx\" (UniqueName: \"kubernetes.io/projected/dca83999-a27c-4d5b-8ab5-09ccbf01f5a6-kube-api-access-stmnx\") pod \"echo-simulator-kserve-64b6f8f96c-dw8g7\" (UID: \"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:23.020814 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:23.020722 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:23.320918 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:23.320887 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dw8g7"] Oct 30 01:47:23.322378 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:47:23.322348 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca83999_a27c_4d5b_8ab5_09ccbf01f5a6.slice/crio-632da302c801138ac239546331cef55fe0efbd8cfeec82fb0f97b746c911b7b5 WatchSource:0}: Error finding container 632da302c801138ac239546331cef55fe0efbd8cfeec82fb0f97b746c911b7b5: Status 404 returned error can't find the container with id 632da302c801138ac239546331cef55fe0efbd8cfeec82fb0f97b746c911b7b5 Oct 30 01:47:23.926600 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:23.926556 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" event={"ID":"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6","Type":"ContainerStarted","Data":"632da302c801138ac239546331cef55fe0efbd8cfeec82fb0f97b746c911b7b5"} Oct 30 01:47:25.352434 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.352392 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj"] Oct 30 01:47:25.359306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.359277 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.380889 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.380854 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj"] Oct 30 01:47:25.388474 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.388436 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 01:47:25.518062 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.518006 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.518062 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.518065 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.518359 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.518176 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.518359 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.518223 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrtq\" (UniqueName: \"kubernetes.io/projected/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-kube-api-access-qrrtq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.518359 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.518326 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.518359 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.518349 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.619504 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.619402 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrtq\" (UniqueName: \"kubernetes.io/projected/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-kube-api-access-qrrtq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.619504 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.619496 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.619768 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.619554 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.619768 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.619644 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.619768 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.619668 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.619768 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.619707 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.620015 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.619989 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.620110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.620081 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.620156 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.620117 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.622216 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.622184 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.622721 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.622702 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.646976 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.646935 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrtq\" (UniqueName: \"kubernetes.io/projected/ed9fd4ef-0aab-454d-b95f-83cbcb4fe367-kube-api-access-qrrtq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj\" (UID: \"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.677272 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.677158 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:25.939481 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.939397 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 01:47:25.939481 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.939446 5035 generic.go:356] "Generic (PLEG): container finished" podID="dca83999-a27c-4d5b-8ab5-09ccbf01f5a6" containerID="490b55bde8949054fb4d6f3d6d847a6f0a07ca20a67bde5470765a99be73eee1" exitCode=137 Oct 30 01:47:25.939710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.939480 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" event={"ID":"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6","Type":"ContainerDied","Data":"490b55bde8949054fb4d6f3d6d847a6f0a07ca20a67bde5470765a99be73eee1"} Oct 30 01:47:25.939880 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.939849 5035 scope.go:122] "RemoveContainer" containerID="490b55bde8949054fb4d6f3d6d847a6f0a07ca20a67bde5470765a99be73eee1" Oct 30 01:47:25.986016 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:25.985857 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj"] Oct 30 01:47:25.988358 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:47:25.988314 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded9fd4ef_0aab_454d_b95f_83cbcb4fe367.slice/crio-464051a0779578bec75664cd9d8bfc3e471c879e3f62eed9a1fd62a86e5506e1 WatchSource:0}: Error finding container 464051a0779578bec75664cd9d8bfc3e471c879e3f62eed9a1fd62a86e5506e1: Status 404 returned error can't find the container with id 464051a0779578bec75664cd9d8bfc3e471c879e3f62eed9a1fd62a86e5506e1 Oct 30 01:47:26.021076 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:26.021024 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:26.947654 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:26.947622 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 01:47:26.948156 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:26.947743 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" event={"ID":"dca83999-a27c-4d5b-8ab5-09ccbf01f5a6","Type":"ContainerStarted","Data":"6531d4c8636e2bcbe6427119cd7df40c0cbc464f0a35feee4a7bad1c4aa27a75"} Oct 30 01:47:26.949598 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:26.949557 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" event={"ID":"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367","Type":"ContainerStarted","Data":"0222eb56b579dbdd5ad7ad671d069df01be3f4c5ddea098f72fd32e7ec609c0a"} Oct 30 01:47:26.949764 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:26.949609 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" event={"ID":"ed9fd4ef-0aab-454d-b95f-83cbcb4fe367","Type":"ContainerStarted","Data":"464051a0779578bec75664cd9d8bfc3e471c879e3f62eed9a1fd62a86e5506e1"} Oct 30 01:47:27.003262 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:27.003174 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" podStartSLOduration=3.155331483 podStartE2EDuration="5.003150897s" totalImagesPullingTime="1.847819414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:27.000958663 +0000 UTC m=+1125.503890908" watchObservedRunningTime="2026-10-30 01:47:27.003150897 +0000 UTC m=+1125.506083118" Oct 30 01:47:27.021613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:27.021569 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:27.048898 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:27.048840 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podStartSLOduration=1.864043654 podStartE2EDuration="2.048824146s" totalImagesPullingTime="184.780492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:27.04796864 +0000 UTC m=+1125.550900859" watchObservedRunningTime="2026-10-30 01:47:27.048824146 +0000 UTC m=+1125.551756366" Oct 30 01:47:27.362206 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:27.362139 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:27.380734 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:27.380696 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:27.678228 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:27.678132 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:27.966044 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:27.966011 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4dsxw" Oct 30 01:47:28.921355 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:28.921305 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:28.934491 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:28.934455 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:28.972518 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:28.972483 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-v5zns" Oct 30 01:47:33.021552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:33.021485 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:33.034603 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:33.034564 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:34.012218 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:34.012179 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" Oct 30 01:47:35.678208 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:35.678173 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:35.691265 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:35.691226 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:36.010781 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:36.010749 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:47:47.628987 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:47:47.628940 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:47:55.226833 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.226786 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2"] Oct 30 01:47:55.230984 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.230957 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.262649 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.262613 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2"] Oct 30 01:47:55.302883 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.302838 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qblrj\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-kube-api-access-qblrj\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.302883 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.302887 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.303104 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.302976 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.303147 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.303106 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.303181 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.303146 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.303181 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.303164 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.303243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.303203 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.303280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.303246 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.303315 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.303283 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404158 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404110 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qblrj\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-kube-api-access-qblrj\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404158 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404158 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404392 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404189 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404392 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404269 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404392 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404305 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404392 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404330 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404392 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404369 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404712 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404501 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404712 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404569 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404872 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404847 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.404916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404882 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.405016 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.404992 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.405218 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.405049 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.405336 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.405286 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.406781 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.406759 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.407018 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.406998 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.429809 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.429768 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.429971 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.429862 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qblrj\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-kube-api-access-qblrj\") pod \"maas-default-gateway-openshift-default-75d7c785f-gjlm2\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.545643 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.545521 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:55.873467 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.873431 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2"] Oct 30 01:47:55.876409 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:47:55.876376 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587cfc66_c33f_40b7_b2a0_19639bfa8aef.slice/crio-e30182a74f686a9ba253ddbdd15ec5e49ec02214f5ff21352a8114d30a8aa9bd WatchSource:0}: Error finding container e30182a74f686a9ba253ddbdd15ec5e49ec02214f5ff21352a8114d30a8aa9bd: Status 404 returned error can't find the container with id e30182a74f686a9ba253ddbdd15ec5e49ec02214f5ff21352a8114d30a8aa9bd Oct 30 01:47:55.878712 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.878677 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:55.878799 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.878749 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:55.878799 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:55.878778 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:56.108321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:56.108281 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" event={"ID":"587cfc66-c33f-40b7-b2a0-19639bfa8aef","Type":"ContainerStarted","Data":"1596114c23219865b5e6792b97e49f2b40f1cfbb160514854e28aeb355c99868"} Oct 30 01:47:56.108506 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:56.108337 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" event={"ID":"587cfc66-c33f-40b7-b2a0-19639bfa8aef","Type":"ContainerStarted","Data":"e30182a74f686a9ba253ddbdd15ec5e49ec02214f5ff21352a8114d30a8aa9bd"} Oct 30 01:47:56.156205 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:56.156095 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podStartSLOduration=1.156078707 podStartE2EDuration="1.156078707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:56.155295821 +0000 UTC m=+1154.658228065" watchObservedRunningTime="2026-10-30 01:47:56.156078707 +0000 UTC m=+1154.659010927" Oct 30 01:47:56.546247 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:56.546195 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:47:56.547940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:56.547901 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gjlm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:47:56.548012 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:56.547982 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 01:47:57.546377 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:57.546331 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gjlm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:47:57.546918 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:57.546397 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 01:47:58.546146 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:58.546101 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gjlm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:47:58.546334 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:58.546164 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 01:47:59.546723 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:59.546686 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gjlm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:47:59.547145 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:47:59.546759 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 01:48:00.546369 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:00.546329 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gjlm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:48:00.546603 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:00.546391 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 01:48:01.546184 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:01.546137 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gjlm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:48:01.546641 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:01.546199 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 01:48:02.546586 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:02.546517 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gjlm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:48:02.547023 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:02.546608 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 01:48:04.270833 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:04.270794 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:48:04.271357 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:04.271334 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:48:04.272314 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:04.272292 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 01:48:10.279979 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.279940 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r"] Oct 30 01:48:10.283836 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.283805 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.308361 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.308322 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r"] Oct 30 01:48:10.382449 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.382404 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.382449 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.382451 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.382741 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.382567 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.382741 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.382667 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.382741 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.382694 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.382864 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.382746 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.382864 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.382774 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.382864 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.382797 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2qq2\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-kube-api-access-x2qq2\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.382959 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.382870 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.483954 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.483902 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.483954 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.483959 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.483992 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484017 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484035 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484055 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484078 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484105 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2qq2\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-kube-api-access-x2qq2\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484146 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484407 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484484 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484567 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484746 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484601 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.484852 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.484827 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.486637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.486617 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.486877 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.486856 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.509334 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.509294 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.509601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.509578 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2qq2\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-kube-api-access-x2qq2\") pod \"maas-default-gateway-openshift-default-75d7c785f-46t2r\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.597816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.597717 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:10.913897 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.913869 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r"] Oct 30 01:48:10.915762 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:10.915736 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49859e7_272b_41fb_9950_c85bf8cdb9ea.slice/crio-47dbf8658948ca2bb011e3fcbf9621ef4d89b30b6e2702443e722a09b377573a WatchSource:0}: Error finding container 47dbf8658948ca2bb011e3fcbf9621ef4d89b30b6e2702443e722a09b377573a: Status 404 returned error can't find the container with id 47dbf8658948ca2bb011e3fcbf9621ef4d89b30b6e2702443e722a09b377573a Oct 30 01:48:10.918086 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.918056 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:10.918152 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.918128 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:10.918187 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:10.918155 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:11.187862 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:11.187769 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" event={"ID":"b49859e7-272b-41fb-9950-c85bf8cdb9ea","Type":"ContainerStarted","Data":"e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a"} Oct 30 01:48:11.187862 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:11.187826 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" event={"ID":"b49859e7-272b-41fb-9950-c85bf8cdb9ea","Type":"ContainerStarted","Data":"47dbf8658948ca2bb011e3fcbf9621ef4d89b30b6e2702443e722a09b377573a"} Oct 30 01:48:11.236518 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:11.236453 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podStartSLOduration=1.23643801 podStartE2EDuration="1.23643801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:11.235399052 +0000 UTC m=+1169.738331272" watchObservedRunningTime="2026-10-30 01:48:11.23643801 +0000 UTC m=+1169.739370229" Oct 30 01:48:11.598065 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:11.598018 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:11.599827 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:11.599798 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-46t2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:11.599967 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:11.599855 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 01:48:12.598818 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:12.598781 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-46t2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:12.599233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:12.598838 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 01:48:13.598762 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:13.598725 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-46t2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:13.598954 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:13.598786 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 01:48:14.599128 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:14.599087 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-46t2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:14.599581 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:14.599145 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 01:48:15.426539 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.426496 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd"] Oct 30 01:48:15.431179 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.431147 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.447123 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.447087 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bc6b04f0-openshift-default-dockercfg-jgf2d\"" Oct 30 01:48:15.463762 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.463718 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd"] Oct 30 01:48:15.534101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.534061 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-certs\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.534101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.534103 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-podinfo\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.534343 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.534132 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-credential-socket\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.534343 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.534202 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46b78c8-d377-4f5e-a828-759559b2ed13-istiod-ca-cert\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.534343 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.534264 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-token\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.534343 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.534296 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcmrj\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-kube-api-access-tcmrj\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.534593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.534433 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-envoy\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.534593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.534510 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-data\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.534593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.534572 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-socket\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.599158 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.599120 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-46t2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:15.599586 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.599190 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 01:48:15.636178 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636134 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-envoy\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.636356 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636197 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-data\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.636356 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636226 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-socket\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.636356 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636315 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-certs\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.636356 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636347 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-podinfo\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.636577 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636389 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-credential-socket\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.636577 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636421 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46b78c8-d377-4f5e-a828-759559b2ed13-istiod-ca-cert\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.636577 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636451 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-token\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.636577 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636481 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcmrj\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-kube-api-access-tcmrj\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.636944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.636913 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-socket\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.637099 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.637075 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-certs\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.637789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.637355 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-credential-socket\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.637789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.637609 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-data\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.638073 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.637864 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46b78c8-d377-4f5e-a828-759559b2ed13-istiod-ca-cert\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.642576 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.642511 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-podinfo\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.643042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.643012 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-envoy\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.673556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.671307 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcmrj\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-kube-api-access-tcmrj\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.700271 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.700244 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-token\") pod \"e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.748954 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.748901 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:15.799456 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.799419 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg"] Oct 30 01:48:15.803815 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.803790 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.828267 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.826658 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg"] Oct 30 01:48:15.839330 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.839191 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-data\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.839330 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.839251 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-certs\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.839330 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.839333 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hpq\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-kube-api-access-s7hpq\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.839661 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.839373 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-podinfo\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.839661 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.839409 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-envoy\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.839661 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.839434 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istiod-ca-cert\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.839661 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.839498 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-socket\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.839661 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.839521 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-credential-socket\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.839661 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.839589 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-token\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.902293 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.900963 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7664a237-gw-openshift-default-dockercfg-5k876\"" Oct 30 01:48:15.941357 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941276 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-certs\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.941568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941433 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hpq\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-kube-api-access-s7hpq\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.941568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941482 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-podinfo\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.941702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941579 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-envoy\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.941702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941608 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istiod-ca-cert\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.941702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941679 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-socket\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.941702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941684 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-certs\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.941702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941702 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-credential-socket\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.941978 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941738 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-token\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.941978 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.941805 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-data\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.942929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.942183 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-data\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.942929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.942385 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-socket\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.942929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.942456 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-credential-socket\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.942929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.942897 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istiod-ca-cert\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.943831 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.943810 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-envoy\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.944388 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.944364 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-podinfo\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.966105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.966025 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hpq\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-kube-api-access-s7hpq\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:15.966242 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:15.966152 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-token\") pod \"e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:16.119063 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.119034 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd"] Oct 30 01:48:16.121899 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:16.121868 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46b78c8_d377_4f5e_a828_759559b2ed13.slice/crio-6004d0d685688a9cdfff16467f82cd07ea943a785694106968f28531fc3d9936 WatchSource:0}: Error finding container 6004d0d685688a9cdfff16467f82cd07ea943a785694106968f28531fc3d9936: Status 404 returned error can't find the container with id 6004d0d685688a9cdfff16467f82cd07ea943a785694106968f28531fc3d9936 Oct 30 01:48:16.124290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.124256 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:16.124395 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.124320 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:16.124395 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.124351 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:16.131997 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.131903 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:16.230900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.230750 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" event={"ID":"d46b78c8-d377-4f5e-a828-759559b2ed13","Type":"ContainerStarted","Data":"6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42"} Oct 30 01:48:16.230900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.230819 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" event={"ID":"d46b78c8-d377-4f5e-a828-759559b2ed13","Type":"ContainerStarted","Data":"6004d0d685688a9cdfff16467f82cd07ea943a785694106968f28531fc3d9936"} Oct 30 01:48:16.293442 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.291780 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" podStartSLOduration=1.291757978 podStartE2EDuration="1.291757978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:16.289069403 +0000 UTC m=+1174.792001626" watchObservedRunningTime="2026-10-30 01:48:16.291757978 +0000 UTC m=+1174.794690198" Oct 30 01:48:16.473618 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.467695 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg"] Oct 30 01:48:16.498982 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:16.498933 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40709a04_b185_4f5f_bdf3_294c6e3b4eed.slice/crio-757c273ee79b8c5f034c508af976cea86929bb47f7c057c5558ae87dd5f0b3c8 WatchSource:0}: Error finding container 757c273ee79b8c5f034c508af976cea86929bb47f7c057c5558ae87dd5f0b3c8: Status 404 returned error can't find the container with id 757c273ee79b8c5f034c508af976cea86929bb47f7c057c5558ae87dd5f0b3c8 Oct 30 01:48:16.501364 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.501331 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:16.501444 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.501400 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:16.501444 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.501428 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:16.598934 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.598893 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-46t2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:16.599092 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.598945 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 01:48:16.749759 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:16.749671 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:17.238753 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:17.238707 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" event={"ID":"40709a04-b185-4f5f-bdf3-294c6e3b4eed","Type":"ContainerStarted","Data":"c053d28bf5794d6e7fdb3a34f5351bc913e4e90ec7df7a3910f10fa6d1417a37"} Oct 30 01:48:17.238962 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:17.238768 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" event={"ID":"40709a04-b185-4f5f-bdf3-294c6e3b4eed","Type":"ContainerStarted","Data":"757c273ee79b8c5f034c508af976cea86929bb47f7c057c5558ae87dd5f0b3c8"} Oct 30 01:48:17.288248 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:17.288186 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" podStartSLOduration=2.2881699380000002 podStartE2EDuration="2.288169938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:17.285338614 +0000 UTC m=+1175.788270834" watchObservedRunningTime="2026-10-30 01:48:17.288169938 +0000 UTC m=+1175.791102159" Oct 30 01:48:17.598740 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:17.598639 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-46t2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:48:17.598740 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:17.598715 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 01:48:17.631716 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:48:17.631679 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:48:17.755501 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:17.755466 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:18.133033 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:18.132990 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:18.138995 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:18.138963 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:18.244015 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:18.243980 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:18.244204 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:18.244036 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:18.245077 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:18.245051 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:18.245203 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:18.245187 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:19.351034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:19.351002 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:19.351638 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:19.351291 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:19.352159 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:19.352141 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 01:48:21.473625 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.473588 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s"] Oct 30 01:48:21.477710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.477687 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:21.500905 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.500871 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-61b6469a\"" Oct 30 01:48:21.516882 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.516847 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s"] Oct 30 01:48:21.520182 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.520147 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74"] Oct 30 01:48:21.524050 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.524025 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:21.544405 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.544371 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-61b6469a-dockercfg-xh6tl\"" Oct 30 01:48:21.546694 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.546662 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74"] Oct 30 01:48:21.608222 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.608187 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjhr\" (UniqueName: \"kubernetes.io/projected/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-kube-api-access-zrjhr\") pod \"payload-processing-e2e-ait-61b6469a-d856545d4-2z75s\" (UID: \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\") " pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:21.608399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.608257 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fjp\" (UniqueName: \"kubernetes.io/projected/2f78bad0-c46b-4f70-a56f-42173f06469c-kube-api-access-x6fjp\") pod \"payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74\" (UID: \"2f78bad0-c46b-4f70-a56f-42173f06469c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:21.608399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.608342 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-61b6469a-d856545d4-2z75s\" (UID: \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\") " pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:21.608399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.608393 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f78bad0-c46b-4f70-a56f-42173f06469c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74\" (UID: \"2f78bad0-c46b-4f70-a56f-42173f06469c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:21.709047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.709002 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjhr\" (UniqueName: \"kubernetes.io/projected/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-kube-api-access-zrjhr\") pod \"payload-processing-e2e-ait-61b6469a-d856545d4-2z75s\" (UID: \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\") " pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:21.709241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.709160 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fjp\" (UniqueName: \"kubernetes.io/projected/2f78bad0-c46b-4f70-a56f-42173f06469c-kube-api-access-x6fjp\") pod \"payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74\" (UID: \"2f78bad0-c46b-4f70-a56f-42173f06469c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:21.709241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.709219 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-61b6469a-d856545d4-2z75s\" (UID: \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\") " pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:21.709354 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.709272 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f78bad0-c46b-4f70-a56f-42173f06469c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74\" (UID: \"2f78bad0-c46b-4f70-a56f-42173f06469c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:21.710049 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.710023 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f78bad0-c46b-4f70-a56f-42173f06469c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74\" (UID: \"2f78bad0-c46b-4f70-a56f-42173f06469c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:21.710176 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.710082 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-61b6469a-d856545d4-2z75s\" (UID: \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\") " pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:21.741756 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.741664 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fjp\" (UniqueName: \"kubernetes.io/projected/2f78bad0-c46b-4f70-a56f-42173f06469c-kube-api-access-x6fjp\") pod \"payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74\" (UID: \"2f78bad0-c46b-4f70-a56f-42173f06469c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:21.742620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.742598 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjhr\" (UniqueName: \"kubernetes.io/projected/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-kube-api-access-zrjhr\") pod \"payload-processing-e2e-ait-61b6469a-d856545d4-2z75s\" (UID: \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\") " pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:21.792542 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.792500 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:21.838973 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:21.838937 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:22.098059 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.098031 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s"] Oct 30 01:48:22.100600 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:22.100561 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6758ea8e_24ef_4f79_9f88_e4b74f4cd8ed.slice/crio-f2fd142e41c7003bc481b607b4769f1d0ea4d2344b0cd89de651c28c9ec4235c WatchSource:0}: Error finding container f2fd142e41c7003bc481b607b4769f1d0ea4d2344b0cd89de651c28c9ec4235c: Status 404 returned error can't find the container with id f2fd142e41c7003bc481b607b4769f1d0ea4d2344b0cd89de651c28c9ec4235c Oct 30 01:48:22.158574 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.158542 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74"] Oct 30 01:48:22.161300 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:22.161270 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f78bad0_c46b_4f70_a56f_42173f06469c.slice/crio-5f0a36086998d4c5adb5ec9ee8f2eda7c517af9032f6721683681626e5871e6c WatchSource:0}: Error finding container 5f0a36086998d4c5adb5ec9ee8f2eda7c517af9032f6721683681626e5871e6c: Status 404 returned error can't find the container with id 5f0a36086998d4c5adb5ec9ee8f2eda7c517af9032f6721683681626e5871e6c Oct 30 01:48:22.269467 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.269422 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" event={"ID":"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed","Type":"ContainerStarted","Data":"f2fd142e41c7003bc481b607b4769f1d0ea4d2344b0cd89de651c28c9ec4235c"} Oct 30 01:48:22.271093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.271054 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" event={"ID":"2f78bad0-c46b-4f70-a56f-42173f06469c","Type":"ContainerStarted","Data":"5f0a36086998d4c5adb5ec9ee8f2eda7c517af9032f6721683681626e5871e6c"} Oct 30 01:48:22.495743 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.495705 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7"] Oct 30 01:48:22.501001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.500979 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:22.519774 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.519746 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7664a237\"" Oct 30 01:48:22.519774 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.519750 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7"] Oct 30 01:48:22.519774 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.519773 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7664a237-dockercfg-cvfrm\"" Oct 30 01:48:22.626184 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.626087 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7\" (UID: \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:22.626414 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.626299 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5ll\" (UniqueName: \"kubernetes.io/projected/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-kube-api-access-lg5ll\") pod \"payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7\" (UID: \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:22.728097 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.728054 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5ll\" (UniqueName: \"kubernetes.io/projected/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-kube-api-access-lg5ll\") pod \"payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7\" (UID: \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:22.728308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.728216 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7\" (UID: \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:22.729129 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.729094 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7\" (UID: \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:22.759851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.759759 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5ll\" (UniqueName: \"kubernetes.io/projected/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-kube-api-access-lg5ll\") pod \"payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7\" (UID: \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:22.815222 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:22.815176 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:23.147109 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:23.147053 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7"] Oct 30 01:48:23.156858 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:23.156820 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968430eb_9bf4_4f33_b8fc_0c8cfe34dafc.slice/crio-98a90a68893497fe3abe5ac84e7b9e28fe4fcdc8b63e4a57c7b8de81ce8a442e WatchSource:0}: Error finding container 98a90a68893497fe3abe5ac84e7b9e28fe4fcdc8b63e4a57c7b8de81ce8a442e: Status 404 returned error can't find the container with id 98a90a68893497fe3abe5ac84e7b9e28fe4fcdc8b63e4a57c7b8de81ce8a442e Oct 30 01:48:23.280337 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:23.280289 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" event={"ID":"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc","Type":"ContainerStarted","Data":"98a90a68893497fe3abe5ac84e7b9e28fe4fcdc8b63e4a57c7b8de81ce8a442e"} Oct 30 01:48:24.658551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.655860 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd"] Oct 30 01:48:24.672395 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.672356 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.688066 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.688033 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd"] Oct 30 01:48:24.694939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.694881 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6c3505dc-openshift-default-dockercfg-hhsc2\"" Oct 30 01:48:24.755549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.755331 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-istio-token\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.755549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.755398 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1787c98c-df0e-41b6-bef6-e414485965d4-istiod-ca-cert\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.755815 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.755517 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv88w\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-kube-api-access-hv88w\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.755815 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.755623 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-certs\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.755815 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.755754 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1787c98c-df0e-41b6-bef6-e414485965d4-istio-podinfo\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.755815 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.755787 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-socket\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.756093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.755823 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-data\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.756093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.755869 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-envoy\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.756093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.755946 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-credential-socket\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.857694 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.857651 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-istio-token\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.857859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.857709 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1787c98c-df0e-41b6-bef6-e414485965d4-istiod-ca-cert\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.857859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.857739 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv88w\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-kube-api-access-hv88w\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.857859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.857760 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-certs\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.857859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.857808 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1787c98c-df0e-41b6-bef6-e414485965d4-istio-podinfo\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.857859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.857834 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-socket\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.858101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.857862 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-data\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.858101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.857902 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-envoy\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.858101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.857954 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-credential-socket\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.859240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.858344 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-credential-socket\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.859240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.858540 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-certs\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.859240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.858939 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-data\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.859240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.859167 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1787c98c-df0e-41b6-bef6-e414485965d4-istiod-ca-cert\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.859240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.859198 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-socket\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.861103 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.861077 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-envoy\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.861103 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.861089 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1787c98c-df0e-41b6-bef6-e414485965d4-istio-podinfo\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.884594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.884554 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv88w\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-kube-api-access-hv88w\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.884815 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.884341 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-istio-token\") pod \"e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:24.942077 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.942038 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v"] Oct 30 01:48:24.948149 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.948119 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:24.970571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.965776 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-14b53b75\"" Oct 30 01:48:24.970571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.965827 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-14b53b75-dockercfg-xfkdt\"" Oct 30 01:48:24.970571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.965940 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v"] Oct 30 01:48:24.993233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:24.993207 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:25.062470 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.062422 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27420bb1-897c-4ce7-933f-c95780592774-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v\" (UID: \"27420bb1-897c-4ce7-933f-c95780592774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:25.062694 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.062484 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccj7c\" (UniqueName: \"kubernetes.io/projected/27420bb1-897c-4ce7-933f-c95780592774-kube-api-access-ccj7c\") pod \"payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v\" (UID: \"27420bb1-897c-4ce7-933f-c95780592774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:25.164565 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.164453 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27420bb1-897c-4ce7-933f-c95780592774-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v\" (UID: \"27420bb1-897c-4ce7-933f-c95780592774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:25.164565 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.164553 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccj7c\" (UniqueName: \"kubernetes.io/projected/27420bb1-897c-4ce7-933f-c95780592774-kube-api-access-ccj7c\") pod \"payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v\" (UID: \"27420bb1-897c-4ce7-933f-c95780592774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:25.165220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.165192 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27420bb1-897c-4ce7-933f-c95780592774-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v\" (UID: \"27420bb1-897c-4ce7-933f-c95780592774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:25.189457 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.189420 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccj7c\" (UniqueName: \"kubernetes.io/projected/27420bb1-897c-4ce7-933f-c95780592774-kube-api-access-ccj7c\") pod \"payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v\" (UID: \"27420bb1-897c-4ce7-933f-c95780592774\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:25.298554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.280777 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:25.310958 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.310709 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" event={"ID":"2f78bad0-c46b-4f70-a56f-42173f06469c","Type":"ContainerStarted","Data":"1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d"} Oct 30 01:48:25.311902 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.311861 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:25.314090 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.313443 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" event={"ID":"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed","Type":"ContainerStarted","Data":"133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061"} Oct 30 01:48:25.314090 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.313572 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:25.357821 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.357733 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" podStartSLOduration=1.614716197 podStartE2EDuration="4.35770957s" totalImagesPullingTime="2.742993373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:25.354774996 +0000 UTC m=+1183.857707222" watchObservedRunningTime="2026-10-30 01:48:25.35770957 +0000 UTC m=+1183.860641790" Oct 30 01:48:25.414900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.414754 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" podStartSLOduration=1.692749254 podStartE2EDuration="4.414734741s" totalImagesPullingTime="2.721985487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:25.412030523 +0000 UTC m=+1183.914962744" watchObservedRunningTime="2026-10-30 01:48:25.414734741 +0000 UTC m=+1183.917666962" Oct 30 01:48:25.420668 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.420628 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd"] Oct 30 01:48:25.425598 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:25.425561 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1787c98c_df0e_41b6_bef6_e414485965d4.slice/crio-a32e1b3b9e449457d84e5ada57643f31d66466c66ce3ff89da972545a9d5f98d WatchSource:0}: Error finding container a32e1b3b9e449457d84e5ada57643f31d66466c66ce3ff89da972545a9d5f98d: Status 404 returned error can't find the container with id a32e1b3b9e449457d84e5ada57643f31d66466c66ce3ff89da972545a9d5f98d Oct 30 01:48:25.428974 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.428933 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:25.429100 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.429031 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:25.429100 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.429077 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:25.622708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:25.622654 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v"] Oct 30 01:48:25.627200 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:25.627157 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27420bb1_897c_4ce7_933f_c95780592774.slice/crio-a44a12cf353b6974495e2ce6fe2cf590d65f5cb82e90ed38305dc09a7ed8145a WatchSource:0}: Error finding container a44a12cf353b6974495e2ce6fe2cf590d65f5cb82e90ed38305dc09a7ed8145a: Status 404 returned error can't find the container with id a44a12cf353b6974495e2ce6fe2cf590d65f5cb82e90ed38305dc09a7ed8145a Oct 30 01:48:26.321147 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.321101 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" event={"ID":"1787c98c-df0e-41b6-bef6-e414485965d4","Type":"ContainerStarted","Data":"a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd"} Oct 30 01:48:26.321721 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.321199 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" event={"ID":"1787c98c-df0e-41b6-bef6-e414485965d4","Type":"ContainerStarted","Data":"a32e1b3b9e449457d84e5ada57643f31d66466c66ce3ff89da972545a9d5f98d"} Oct 30 01:48:26.322958 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.322924 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" event={"ID":"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc","Type":"ContainerStarted","Data":"89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4"} Oct 30 01:48:26.323084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.322988 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:26.324600 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.324575 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" event={"ID":"27420bb1-897c-4ce7-933f-c95780592774","Type":"ContainerStarted","Data":"2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423"} Oct 30 01:48:26.324722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.324623 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" event={"ID":"27420bb1-897c-4ce7-933f-c95780592774","Type":"ContainerStarted","Data":"a44a12cf353b6974495e2ce6fe2cf590d65f5cb82e90ed38305dc09a7ed8145a"} Oct 30 01:48:26.324722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.324703 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:26.393072 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.393003 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" podStartSLOduration=2.392983076 podStartE2EDuration="2.392983076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:26.392244997 +0000 UTC m=+1184.895177220" watchObservedRunningTime="2026-10-30 01:48:26.392983076 +0000 UTC m=+1184.895915297" Oct 30 01:48:26.436107 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.436049 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" podStartSLOduration=2.436034219 podStartE2EDuration="2.436034219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:26.434644214 +0000 UTC m=+1184.937576436" watchObservedRunningTime="2026-10-30 01:48:26.436034219 +0000 UTC m=+1184.938966439" Oct 30 01:48:26.994494 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:26.994438 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:27.000415 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:27.000384 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:27.053832 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:27.053769 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" podStartSLOduration=2.6048877089999998 podStartE2EDuration="5.053746954s" totalImagesPullingTime="2.448859245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:26.488738906 +0000 UTC m=+1184.991671129" watchObservedRunningTime="2026-10-30 01:48:27.053746954 +0000 UTC m=+1185.556679174" Oct 30 01:48:27.331222 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:27.331126 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:27.332296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:27.332273 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:30.419556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.419503 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj"] Oct 30 01:48:30.423583 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.423560 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:30.445636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.445597 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj"] Oct 30 01:48:30.455362 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.455319 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6c3505dc\"" Oct 30 01:48:30.531494 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.531443 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq"] Oct 30 01:48:30.536323 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.536294 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq"] Oct 30 01:48:30.536465 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.536425 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:30.549807 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.549715 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa99545f-be08-4953-8537-54935ebcceb3-plugins-config-volume\") pod \"payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj\" (UID: \"aa99545f-be08-4953-8537-54935ebcceb3\") " pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:30.549997 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.549969 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2dv8\" (UniqueName: \"kubernetes.io/projected/aa99545f-be08-4953-8537-54935ebcceb3-kube-api-access-l2dv8\") pod \"payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj\" (UID: \"aa99545f-be08-4953-8537-54935ebcceb3\") " pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:30.553574 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.552755 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6c3505dc-dockercfg-65c5s\"" Oct 30 01:48:30.651246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.651192 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2dv8\" (UniqueName: \"kubernetes.io/projected/aa99545f-be08-4953-8537-54935ebcceb3-kube-api-access-l2dv8\") pod \"payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj\" (UID: \"aa99545f-be08-4953-8537-54935ebcceb3\") " pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:30.651469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.651254 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mptf\" (UniqueName: \"kubernetes.io/projected/604332d8-bef6-4ba2-85d9-65fa46b15d79-kube-api-access-2mptf\") pod \"payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq\" (UID: \"604332d8-bef6-4ba2-85d9-65fa46b15d79\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:30.651469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.651371 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604332d8-bef6-4ba2-85d9-65fa46b15d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq\" (UID: \"604332d8-bef6-4ba2-85d9-65fa46b15d79\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:30.651469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.651408 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa99545f-be08-4953-8537-54935ebcceb3-plugins-config-volume\") pod \"payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj\" (UID: \"aa99545f-be08-4953-8537-54935ebcceb3\") " pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:30.652697 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.652675 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa99545f-be08-4953-8537-54935ebcceb3-plugins-config-volume\") pod \"payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj\" (UID: \"aa99545f-be08-4953-8537-54935ebcceb3\") " pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:30.713124 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.713076 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2dv8\" (UniqueName: \"kubernetes.io/projected/aa99545f-be08-4953-8537-54935ebcceb3-kube-api-access-l2dv8\") pod \"payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj\" (UID: \"aa99545f-be08-4953-8537-54935ebcceb3\") " pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:30.739437 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.739396 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:30.752797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.752753 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mptf\" (UniqueName: \"kubernetes.io/projected/604332d8-bef6-4ba2-85d9-65fa46b15d79-kube-api-access-2mptf\") pod \"payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq\" (UID: \"604332d8-bef6-4ba2-85d9-65fa46b15d79\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:30.752920 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.752874 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604332d8-bef6-4ba2-85d9-65fa46b15d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq\" (UID: \"604332d8-bef6-4ba2-85d9-65fa46b15d79\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:30.753810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.753784 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604332d8-bef6-4ba2-85d9-65fa46b15d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq\" (UID: \"604332d8-bef6-4ba2-85d9-65fa46b15d79\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:30.794552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.794306 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mptf\" (UniqueName: \"kubernetes.io/projected/604332d8-bef6-4ba2-85d9-65fa46b15d79-kube-api-access-2mptf\") pod \"payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq\" (UID: \"604332d8-bef6-4ba2-85d9-65fa46b15d79\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:30.856191 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:30.856156 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:31.116195 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:31.116077 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj"] Oct 30 01:48:31.124617 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:31.124578 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa99545f_be08_4953_8537_54935ebcceb3.slice/crio-28fc09abf76a359fd394605037ff996e94595ceaaf54044f0998e3c3d0f06bb1 WatchSource:0}: Error finding container 28fc09abf76a359fd394605037ff996e94595ceaaf54044f0998e3c3d0f06bb1: Status 404 returned error can't find the container with id 28fc09abf76a359fd394605037ff996e94595ceaaf54044f0998e3c3d0f06bb1 Oct 30 01:48:31.297999 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:31.295301 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq"] Oct 30 01:48:31.359387 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:31.359345 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" event={"ID":"604332d8-bef6-4ba2-85d9-65fa46b15d79","Type":"ContainerStarted","Data":"ab1673224dfffa80e03311a3121601beef7f53c0563787d0dc54af8c7786b58b"} Oct 30 01:48:31.361015 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:31.360984 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" event={"ID":"aa99545f-be08-4953-8537-54935ebcceb3","Type":"ContainerStarted","Data":"1d0749bd1c651886ce36fc66afb4c4f297ec765c03c1779716f4b98da8da0648"} Oct 30 01:48:31.361153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:31.361036 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" event={"ID":"aa99545f-be08-4953-8537-54935ebcceb3","Type":"ContainerStarted","Data":"28fc09abf76a359fd394605037ff996e94595ceaaf54044f0998e3c3d0f06bb1"} Oct 30 01:48:31.361219 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:31.361183 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:31.414570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:31.412826 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" podStartSLOduration=1.412804819 podStartE2EDuration="1.412804819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:31.410009133 +0000 UTC m=+1189.912941354" watchObservedRunningTime="2026-10-30 01:48:31.412804819 +0000 UTC m=+1189.915737039" Oct 30 01:48:32.391722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:32.391670 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" event={"ID":"604332d8-bef6-4ba2-85d9-65fa46b15d79","Type":"ContainerStarted","Data":"9a9a9fafc5ea754397a494ec47fd1acfd0165984ceb890e4c63131fe94f4a929"} Oct 30 01:48:32.392279 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:32.391822 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:32.437596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:32.437521 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" podStartSLOduration=2.437503405 podStartE2EDuration="2.437503405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:32.435063667 +0000 UTC m=+1190.937995888" watchObservedRunningTime="2026-10-30 01:48:32.437503405 +0000 UTC m=+1190.940435624" Oct 30 01:48:33.894559 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:33.894034 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d"] Oct 30 01:48:33.899867 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:33.899833 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:33.900047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:33.899902 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d"] Oct 30 01:48:33.932321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:33.932265 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:48:33.933357 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:33.933329 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:48:33.933634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:33.933617 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:48:33.945404 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:33.945363 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:48:34.035067 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.035024 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.035294 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.035078 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.035294 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.035229 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvtlj\" (UniqueName: \"kubernetes.io/projected/64cb475d-8277-453f-84d3-37a98b7eb8d9-kube-api-access-kvtlj\") pod \"maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.136427 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.136360 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.136655 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.136500 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtlj\" (UniqueName: \"kubernetes.io/projected/64cb475d-8277-453f-84d3-37a98b7eb8d9-kube-api-access-kvtlj\") pod \"maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.136722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.136665 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.137374 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.137351 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.139009 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.138984 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.161487 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.161398 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtlj\" (UniqueName: \"kubernetes.io/projected/64cb475d-8277-453f-84d3-37a98b7eb8d9-kube-api-access-kvtlj\") pod \"maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.251796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.251749 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:34.293724 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.293009 5035 patch_prober.go:29] interesting pod/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:48:34.293724 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.293075 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:48:34.595554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:34.594919 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d"] Oct 30 01:48:34.601705 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:34.601663 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cb475d_8277_453f_84d3_37a98b7eb8d9.slice/crio-1f7b3f03bcc393ecff92285c620a279e83fd33110774df28d21a899e669f16b6 WatchSource:0}: Error finding container 1f7b3f03bcc393ecff92285c620a279e83fd33110774df28d21a899e669f16b6: Status 404 returned error can't find the container with id 1f7b3f03bcc393ecff92285c620a279e83fd33110774df28d21a899e669f16b6 Oct 30 01:48:35.415542 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:35.415491 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" event={"ID":"64cb475d-8277-453f-84d3-37a98b7eb8d9","Type":"ContainerStarted","Data":"1f7b3f03bcc393ecff92285c620a279e83fd33110774df28d21a899e669f16b6"} Oct 30 01:48:36.329176 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:36.329099 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:48:36.336108 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:36.329698 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:48:36.437198 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:36.436157 5035 generic.go:356] "Generic (PLEG): container finished" podID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerID="6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42" exitCode=0 Oct 30 01:48:36.437198 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:36.436350 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" event={"ID":"d46b78c8-d377-4f5e-a828-759559b2ed13","Type":"ContainerDied","Data":"6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42"} Oct 30 01:48:36.437198 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:36.436905 5035 scope.go:122] "RemoveContainer" containerID="6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42" Oct 30 01:48:36.439786 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:36.439749 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:36.440435 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:36.440404 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:36.440590 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:36.440485 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:37.334439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.334406 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:37.334696 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.334489 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:48:37.446096 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.446059 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" event={"ID":"d46b78c8-d377-4f5e-a828-759559b2ed13","Type":"ContainerStarted","Data":"6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32"} Oct 30 01:48:37.500402 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.500307 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp"] Oct 30 01:48:37.508558 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.506018 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.523665 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.523626 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6c3505dc-serving-cert\"" Oct 30 01:48:37.523822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.523692 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 01:48:37.523822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.523744 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xzt6k\"" Oct 30 01:48:37.535562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.529085 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp"] Oct 30 01:48:37.600968 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.599637 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-metrics-certs\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.600968 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.599722 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h82j\" (UniqueName: \"kubernetes.io/projected/5438ad60-7f7b-4778-b56a-37a06740e8ee-kube-api-access-8h82j\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.600968 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.599864 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.600968 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.599917 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5438ad60-7f7b-4778-b56a-37a06740e8ee-openshift-service-ca\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.705551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.705176 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-metrics-certs\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.705551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.705256 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h82j\" (UniqueName: \"kubernetes.io/projected/5438ad60-7f7b-4778-b56a-37a06740e8ee-kube-api-access-8h82j\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.705551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.705385 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.705551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.705427 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5438ad60-7f7b-4778-b56a-37a06740e8ee-openshift-service-ca\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.711825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.706481 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5438ad60-7f7b-4778-b56a-37a06740e8ee-openshift-service-ca\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.711825 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:48:37.707257 5035 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-serving-cert: secret "maas-api-e2e-derive-6c3505dc-serving-cert" not found Oct 30 01:48:37.711825 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:48:37.707364 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls podName:5438ad60-7f7b-4778-b56a-37a06740e8ee nodeName:}" failed. No retries permitted until 2026-10-30 01:48:38.20734175 +0000 UTC m=+1196.710273951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls") pod "maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" (UID: "5438ad60-7f7b-4778-b56a-37a06740e8ee") : secret "maas-api-e2e-derive-6c3505dc-serving-cert" not found Oct 30 01:48:37.711825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.711056 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-metrics-certs\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.733596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.733559 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h82j\" (UniqueName: \"kubernetes.io/projected/5438ad60-7f7b-4778-b56a-37a06740e8ee-kube-api-access-8h82j\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:37.750205 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.750156 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:37.750389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.750243 5035 patch_prober.go:29] interesting pod/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:37.750389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:37.750304 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 01:48:38.211254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:38.211201 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:38.213825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:38.213787 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls\") pod \"maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:38.430896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:38.430841 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:38.455416 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:38.455343 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" event={"ID":"64cb475d-8277-453f-84d3-37a98b7eb8d9","Type":"ContainerStarted","Data":"9b78ad0bac01880b622561e3993f651915a21f85431d242562bbfac4e8229a24"} Oct 30 01:48:38.523472 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:38.523367 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" podStartSLOduration=2.8901352129999998 podStartE2EDuration="5.523342294s" totalImagesPullingTime="2.633207081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:38.519946266 +0000 UTC m=+1197.022878507" watchObservedRunningTime="2026-10-30 01:48:38.523342294 +0000 UTC m=+1197.026274515" Oct 30 01:48:38.749765 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:38.749671 5035 patch_prober.go:29] interesting pod/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:38.749765 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:38.749744 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 01:48:38.791878 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:38.791447 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp"] Oct 30 01:48:39.464498 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:39.464452 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" event={"ID":"5438ad60-7f7b-4778-b56a-37a06740e8ee","Type":"ContainerStarted","Data":"a8a7a2ebb9744919a34c0be2cafa7a5f89be4dcb4451a970b7c359011b55e87a"} Oct 30 01:48:39.492209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:39.491628 5035 generic.go:356] "Generic (PLEG): container finished" podID="64cb475d-8277-453f-84d3-37a98b7eb8d9" containerID="9b78ad0bac01880b622561e3993f651915a21f85431d242562bbfac4e8229a24" exitCode=0 Oct 30 01:48:39.492209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:39.491753 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" event={"ID":"64cb475d-8277-453f-84d3-37a98b7eb8d9","Type":"ContainerDied","Data":"9b78ad0bac01880b622561e3993f651915a21f85431d242562bbfac4e8229a24"} Oct 30 01:48:39.750160 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:39.750064 5035 patch_prober.go:29] interesting pod/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:39.750160 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:39.750140 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 01:48:40.710859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.710821 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:40.749929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.749829 5035 patch_prober.go:29] interesting pod/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:40.750133 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.749921 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 01:48:40.846037 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.845947 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-xks-ca\") pod \"64cb475d-8277-453f-84d3-37a98b7eb8d9\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " Oct 30 01:48:40.846233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.846094 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-openshift-ca\") pod \"64cb475d-8277-453f-84d3-37a98b7eb8d9\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " Oct 30 01:48:40.846233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.846126 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cb475d-8277-453f-84d3-37a98b7eb8d9-kube-api-access-kvtlj\" (UniqueName: \"kubernetes.io/projected/64cb475d-8277-453f-84d3-37a98b7eb8d9-kube-api-access-kvtlj\") pod \"64cb475d-8277-453f-84d3-37a98b7eb8d9\" (UID: \"64cb475d-8277-453f-84d3-37a98b7eb8d9\") " Oct 30 01:48:40.846499 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.846465 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-openshift-ca" pod "64cb475d-8277-453f-84d3-37a98b7eb8d9" (UID: "64cb475d-8277-453f-84d3-37a98b7eb8d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:40.846628 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.846607 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:40.848907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.848869 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-xks-ca" pod "64cb475d-8277-453f-84d3-37a98b7eb8d9" (UID: "64cb475d-8277-453f-84d3-37a98b7eb8d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:40.848907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.848872 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cb475d-8277-453f-84d3-37a98b7eb8d9-kube-api-access-kvtlj" pod "64cb475d-8277-453f-84d3-37a98b7eb8d9" (UID: "64cb475d-8277-453f-84d3-37a98b7eb8d9"). InnerVolumeSpecName "kube-api-access-kvtlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:40.948105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.948056 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvtlj\" (UniqueName: \"kubernetes.io/projected/64cb475d-8277-453f-84d3-37a98b7eb8d9-kube-api-access-kvtlj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:40.948105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:40.948103 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb475d-8277-453f-84d3-37a98b7eb8d9-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:41.507743 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.507576 5035 generic.go:356] "Generic (PLEG): container finished" podID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerID="c053d28bf5794d6e7fdb3a34f5351bc913e4e90ec7df7a3910f10fa6d1417a37" exitCode=0 Oct 30 01:48:41.508551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.508032 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" event={"ID":"40709a04-b185-4f5f-bdf3-294c6e3b4eed","Type":"ContainerDied","Data":"c053d28bf5794d6e7fdb3a34f5351bc913e4e90ec7df7a3910f10fa6d1417a37"} Oct 30 01:48:41.509067 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.508933 5035 scope.go:122] "RemoveContainer" containerID="c053d28bf5794d6e7fdb3a34f5351bc913e4e90ec7df7a3910f10fa6d1417a37" Oct 30 01:48:41.510109 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.509765 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" event={"ID":"64cb475d-8277-453f-84d3-37a98b7eb8d9","Type":"ContainerDied","Data":"1f7b3f03bcc393ecff92285c620a279e83fd33110774df28d21a899e669f16b6"} Oct 30 01:48:41.510109 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.509800 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d" Oct 30 01:48:41.510109 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.509813 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f7b3f03bcc393ecff92285c620a279e83fd33110774df28d21a899e669f16b6" Oct 30 01:48:41.510783 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.510746 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:41.510875 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.510846 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:41.510934 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.510894 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:41.749738 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.749693 5035 patch_prober.go:29] interesting pod/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:41.750146 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.749750 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 01:48:41.796166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.796077 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 01:48:41.798571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.798543 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 01:48:41.818198 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.818169 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:48:41.820695 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:41.820661 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:48:42.420391 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:42.418938 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:48:42.539484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:42.539426 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" event={"ID":"40709a04-b185-4f5f-bdf3-294c6e3b4eed","Type":"ContainerStarted","Data":"041d8e654dda9d24ee13ae205997ab7d249c12bd1acdcd13d9e391ff096f1cc2"} Oct 30 01:48:42.750177 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:42.750139 5035 patch_prober.go:29] interesting pod/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 01:48:42.750627 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:42.750199 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 01:48:43.132254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.132155 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:43.132703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.132668 5035 patch_prober.go:29] interesting pod/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 01:48:43.132753 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.132725 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 01:48:43.413819 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.413724 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:48:43.552267 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.552232 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" event={"ID":"5438ad60-7f7b-4778-b56a-37a06740e8ee","Type":"ContainerStarted","Data":"b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78"} Oct 30 01:48:43.552457 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.552391 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:43.754914 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.754883 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:43.756072 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.756050 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:43.757033 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.757010 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:43.821411 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:43.821335 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" podStartSLOduration=3.199913853 podStartE2EDuration="6.821313968s" totalImagesPullingTime="3.621400115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:43.617486649 +0000 UTC m=+1202.120418881" watchObservedRunningTime="2026-10-30 01:48:43.821313968 +0000 UTC m=+1202.324246188" Oct 30 01:48:44.133203 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:44.133101 5035 patch_prober.go:29] interesting pod/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 01:48:44.133203 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:44.133172 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 01:48:45.132933 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:45.132887 5035 patch_prober.go:29] interesting pod/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 01:48:45.133354 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:45.132947 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 01:48:46.133157 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:46.133113 5035 patch_prober.go:29] interesting pod/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 01:48:46.133627 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:46.133189 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 01:48:47.132815 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:47.132768 5035 patch_prober.go:29] interesting pod/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 01:48:47.133023 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:47.132852 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 01:48:47.647400 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:48:47.647359 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:48:48.141308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:48.141268 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:48.142256 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:48.142227 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:48.142996 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:48.142975 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:49.606104 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:49.606061 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7"] Oct 30 01:48:50.569904 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:50.569868 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:48:52.655007 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.654921 5035 generic.go:356] "Generic (PLEG): container finished" podID="1787c98c-df0e-41b6-bef6-e414485965d4" containerID="a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd" exitCode=0 Oct 30 01:48:52.655007 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.654986 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" event={"ID":"1787c98c-df0e-41b6-bef6-e414485965d4","Type":"ContainerDied","Data":"a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd"} Oct 30 01:48:52.655653 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.655520 5035 scope.go:122] "RemoveContainer" containerID="a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd" Oct 30 01:48:52.657002 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.656971 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:52.657087 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.657040 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:52.657087 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.657073 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:52.840584 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.840453 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d"] Oct 30 01:48:52.843240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.843171 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc6b04f0-lbq7d"] Oct 30 01:48:52.985480 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.985440 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd"] Oct 30 01:48:52.985794 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:52.985724 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" containerID="cri-o://6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32" gracePeriod=30 Oct 30 01:48:53.505225 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:53.505187 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg"] Oct 30 01:48:53.505687 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:53.505653 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" containerID="cri-o://041d8e654dda9d24ee13ae205997ab7d249c12bd1acdcd13d9e391ff096f1cc2" gracePeriod=30 Oct 30 01:48:53.642666 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:53.642626 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64cb475d-8277-453f-84d3-37a98b7eb8d9" path="/var/lib/kubelet/pods/64cb475d-8277-453f-84d3-37a98b7eb8d9/volumes" Oct 30 01:48:53.665814 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:53.665770 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" event={"ID":"1787c98c-df0e-41b6-bef6-e414485965d4","Type":"ContainerStarted","Data":"fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b"} Oct 30 01:48:53.993782 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:53.993744 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:53.995272 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:53.995245 5035 patch_prober.go:29] interesting pod/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:48:53.995359 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:53.995312 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 01:48:54.606702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.606661 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" podUID="968430eb-9bf4-4f33-b8fc-0c8cfe34dafc" containerName="payload-pre-processing" containerID="cri-o://89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4" gracePeriod=25 Oct 30 01:48:54.803553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.801045 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r"] Oct 30 01:48:54.803553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.801862 5035 cpu_manager.go:414] "containerMap: removing container" podUID="64cb475d-8277-453f-84d3-37a98b7eb8d9" containerName="revoke-keys" Oct 30 01:48:54.803553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.801883 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cb475d-8277-453f-84d3-37a98b7eb8d9" containerName="revoke-keys" Oct 30 01:48:54.803553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.802142 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cb475d-8277-453f-84d3-37a98b7eb8d9" containerName="revoke-keys" Oct 30 01:48:54.806243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.806216 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:54.830509 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.830427 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r"] Oct 30 01:48:54.831580 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.831213 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-78d5ff-openshift-default-dockercfg-s66rj\"" Oct 30 01:48:54.860491 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.860392 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s"] Oct 30 01:48:54.917243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.917206 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jh9g\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-kube-api-access-9jh9g\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:54.917637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.917607 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:54.917742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.917655 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:54.917742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.917692 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:54.917742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.917717 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:54.917742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.917740 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:54.917874 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.917786 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:54.917874 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.917838 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:54.917936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:54.917902 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.004572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.004399 5035 patch_prober.go:29] interesting pod/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:48:55.004572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.004478 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 01:48:55.010567 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.009874 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74"] Oct 30 01:48:55.015509 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.015478 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.018989 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019031 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jh9g\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-kube-api-access-9jh9g\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019079 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019114 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019155 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019190 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019219 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019271 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019330 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019397 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.019685 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.020500 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.020232 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.021230 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.021206 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.022366 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.022053 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.022508 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.022313 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.023224 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.023201 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.048651 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.047610 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.048651 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.048013 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jh9g\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-kube-api-access-9jh9g\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.120434 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.120339 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-plugins-config-volume\") pod \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\" (UID: \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\") " Oct 30 01:48:55.120434 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.120381 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-kube-api-access-lg5ll\" (UniqueName: \"kubernetes.io/projected/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-kube-api-access-lg5ll\") pod \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\" (UID: \"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc\") " Oct 30 01:48:55.120753 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.120727 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-plugins-config-volume" pod "968430eb-9bf4-4f33-b8fc-0c8cfe34dafc" (UID: "968430eb-9bf4-4f33-b8fc-0c8cfe34dafc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:55.124317 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.124279 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-kube-api-access-lg5ll" pod "968430eb-9bf4-4f33-b8fc-0c8cfe34dafc" (UID: "968430eb-9bf4-4f33-b8fc-0c8cfe34dafc"). InnerVolumeSpecName "kube-api-access-lg5ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:55.127869 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.127839 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:55.222552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.222218 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:55.222552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.222260 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5ll\" (UniqueName: \"kubernetes.io/projected/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc-kube-api-access-lg5ll\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:55.491605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.489028 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r"] Oct 30 01:48:55.493124 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:48:55.493090 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a7ded3_59ff_4e1c_bb3a_00956659d646.slice/crio-06d352169b10e92993ac9db3c530bc4f37429f2a8d5e94ae34743fa9a6f40f57 WatchSource:0}: Error finding container 06d352169b10e92993ac9db3c530bc4f37429f2a8d5e94ae34743fa9a6f40f57: Status 404 returned error can't find the container with id 06d352169b10e92993ac9db3c530bc4f37429f2a8d5e94ae34743fa9a6f40f57 Oct 30 01:48:55.495921 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.495870 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:55.496022 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.495978 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:55.496022 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.496013 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:55.705880 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.705835 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" event={"ID":"b5a7ded3-59ff-4e1c-bb3a-00956659d646","Type":"ContainerStarted","Data":"f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded"} Oct 30 01:48:55.706087 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.705925 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" event={"ID":"b5a7ded3-59ff-4e1c-bb3a-00956659d646","Type":"ContainerStarted","Data":"06d352169b10e92993ac9db3c530bc4f37429f2a8d5e94ae34743fa9a6f40f57"} Oct 30 01:48:55.707928 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.707889 5035 generic.go:356] "Generic (PLEG): container finished" podID="968430eb-9bf4-4f33-b8fc-0c8cfe34dafc" containerID="89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4" exitCode=0 Oct 30 01:48:55.708095 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.707933 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" event={"ID":"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc","Type":"ContainerDied","Data":"89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4"} Oct 30 01:48:55.708095 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.707967 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" Oct 30 01:48:55.708095 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.707973 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7" event={"ID":"968430eb-9bf4-4f33-b8fc-0c8cfe34dafc","Type":"ContainerDied","Data":"98a90a68893497fe3abe5ac84e7b9e28fe4fcdc8b63e4a57c7b8de81ce8a442e"} Oct 30 01:48:55.708095 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.707997 5035 scope.go:122] "RemoveContainer" containerID="89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4" Oct 30 01:48:55.718479 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.718454 5035 scope.go:122] "RemoveContainer" containerID="89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4" Oct 30 01:48:55.718872 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:48:55.718851 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4\": container with ID starting with 89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4 not found: ID does not exist" containerID="89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4" Oct 30 01:48:55.718945 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.718883 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4"} err="failed to get container status \"89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4\": rpc error: code = NotFound desc = could not find container \"89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4\": container with ID starting with 89737fcb3411c6e5b83d02b39fb1c0069040bc1bad6db1c3d5314c1a6a168ac4 not found: ID does not exist" Oct 30 01:48:55.756046 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.755975 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" podStartSLOduration=1.755955539 podStartE2EDuration="1.755955539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:55.753020988 +0000 UTC m=+1214.255953232" watchObservedRunningTime="2026-10-30 01:48:55.755955539 +0000 UTC m=+1214.258887759" Oct 30 01:48:55.804079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.804036 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7"] Oct 30 01:48:55.806991 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.806954 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7664a237-6fdb9ccb8c-5g9q7"] Oct 30 01:48:55.998660 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.998286 5035 patch_prober.go:29] interesting pod/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:48:55.998660 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:55.998387 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 01:48:56.129051 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:56.129009 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:56.135191 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:56.135160 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:56.716056 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:56.716023 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:56.717112 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:56.717090 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:48:56.927613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:56.927572 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v"] Oct 30 01:48:56.998671 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:56.996114 5035 patch_prober.go:29] interesting pod/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:48:56.998671 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:56.996173 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 01:48:57.332052 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:57.331953 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:57.642835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:57.642748 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="968430eb-9bf4-4f33-b8fc-0c8cfe34dafc" path="/var/lib/kubelet/pods/968430eb-9bf4-4f33-b8fc-0c8cfe34dafc/volumes" Oct 30 01:48:57.994969 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:57.994922 5035 patch_prober.go:29] interesting pod/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:48:57.995595 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:57.995561 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 01:48:58.322424 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.322394 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:58.355691 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.355636 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-token\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-token\") pod \"d46b78c8-d377-4f5e-a828-759559b2ed13\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " Oct 30 01:48:58.355691 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.355698 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-data\") pod \"d46b78c8-d377-4f5e-a828-759559b2ed13\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " Oct 30 01:48:58.356030 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.355733 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46b78c8-d377-4f5e-a828-759559b2ed13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46b78c8-d377-4f5e-a828-759559b2ed13-istiod-ca-cert\") pod \"d46b78c8-d377-4f5e-a828-759559b2ed13\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " Oct 30 01:48:58.356030 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.355829 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-kube-api-access-tcmrj\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-kube-api-access-tcmrj\") pod \"d46b78c8-d377-4f5e-a828-759559b2ed13\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " Oct 30 01:48:58.356030 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.355898 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-podinfo\") pod \"d46b78c8-d377-4f5e-a828-759559b2ed13\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " Oct 30 01:48:58.356030 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.355946 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-envoy\") pod \"d46b78c8-d377-4f5e-a828-759559b2ed13\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " Oct 30 01:48:58.356030 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.355977 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-socket\") pod \"d46b78c8-d377-4f5e-a828-759559b2ed13\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " Oct 30 01:48:58.356030 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.355985 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-data" pod "d46b78c8-d377-4f5e-a828-759559b2ed13" (UID: "d46b78c8-d377-4f5e-a828-759559b2ed13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.356339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356100 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-credential-socket\") pod \"d46b78c8-d377-4f5e-a828-759559b2ed13\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " Oct 30 01:48:58.356339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356218 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46b78c8-d377-4f5e-a828-759559b2ed13-istiod-ca-cert" pod "d46b78c8-d377-4f5e-a828-759559b2ed13" (UID: "d46b78c8-d377-4f5e-a828-759559b2ed13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:58.356339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356258 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-socket" pod "d46b78c8-d377-4f5e-a828-759559b2ed13" (UID: "d46b78c8-d377-4f5e-a828-759559b2ed13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.356339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356290 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-certs\") pod \"d46b78c8-d377-4f5e-a828-759559b2ed13\" (UID: \"d46b78c8-d377-4f5e-a828-759559b2ed13\") " Oct 30 01:48:58.356572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356493 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-credential-socket" pod "d46b78c8-d377-4f5e-a828-759559b2ed13" (UID: "d46b78c8-d377-4f5e-a828-759559b2ed13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.356683 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356660 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-certs" pod "d46b78c8-d377-4f5e-a828-759559b2ed13" (UID: "d46b78c8-d377-4f5e-a828-759559b2ed13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.356740 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356716 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.356740 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356737 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46b78c8-d377-4f5e-a828-759559b2ed13-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.356846 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356746 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.356846 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.356756 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.358472 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.358437 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-envoy" pod "d46b78c8-d377-4f5e-a828-759559b2ed13" (UID: "d46b78c8-d377-4f5e-a828-759559b2ed13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.358636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.358465 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-token" pod "d46b78c8-d377-4f5e-a828-759559b2ed13" (UID: "d46b78c8-d377-4f5e-a828-759559b2ed13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:58.358732 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.358714 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-podinfo" pod "d46b78c8-d377-4f5e-a828-759559b2ed13" (UID: "d46b78c8-d377-4f5e-a828-759559b2ed13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:48:58.358814 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.358794 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-kube-api-access-tcmrj" pod "d46b78c8-d377-4f5e-a828-759559b2ed13" (UID: "d46b78c8-d377-4f5e-a828-759559b2ed13"). InnerVolumeSpecName "kube-api-access-tcmrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:58.458192 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.458145 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.458192 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.458184 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.458192 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.458197 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcmrj\" (UniqueName: \"kubernetes.io/projected/d46b78c8-d377-4f5e-a828-759559b2ed13-kube-api-access-tcmrj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.458457 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.458212 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.458457 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.458225 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46b78c8-d377-4f5e-a828-759559b2ed13-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.732568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.732267 5035 generic.go:356] "Generic (PLEG): container finished" podID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerID="041d8e654dda9d24ee13ae205997ab7d249c12bd1acdcd13d9e391ff096f1cc2" exitCode=0 Oct 30 01:48:58.732568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.732394 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" event={"ID":"40709a04-b185-4f5f-bdf3-294c6e3b4eed","Type":"ContainerDied","Data":"041d8e654dda9d24ee13ae205997ab7d249c12bd1acdcd13d9e391ff096f1cc2"} Oct 30 01:48:58.732568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.732451 5035 scope.go:122] "RemoveContainer" containerID="c053d28bf5794d6e7fdb3a34f5351bc913e4e90ec7df7a3910f10fa6d1417a37" Oct 30 01:48:58.735576 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.735411 5035 generic.go:356] "Generic (PLEG): container finished" podID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerID="6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32" exitCode=0 Oct 30 01:48:58.735576 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.735468 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" Oct 30 01:48:58.735576 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.735572 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" event={"ID":"d46b78c8-d377-4f5e-a828-759559b2ed13","Type":"ContainerDied","Data":"6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32"} Oct 30 01:48:58.735807 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.735614 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd" event={"ID":"d46b78c8-d377-4f5e-a828-759559b2ed13","Type":"ContainerDied","Data":"6004d0d685688a9cdfff16467f82cd07ea943a785694106968f28531fc3d9936"} Oct 30 01:48:58.753081 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.753051 5035 scope.go:122] "RemoveContainer" containerID="6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32" Oct 30 01:48:58.763714 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.763674 5035 scope.go:122] "RemoveContainer" containerID="6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42" Oct 30 01:48:58.775074 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.775049 5035 scope.go:122] "RemoveContainer" containerID="6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32" Oct 30 01:48:58.775501 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:48:58.775472 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32\": container with ID starting with 6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32 not found: ID does not exist" containerID="6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32" Oct 30 01:48:58.775612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.775512 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32"} err="failed to get container status \"6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32\": rpc error: code = NotFound desc = could not find container \"6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32\": container with ID starting with 6875dd4effcd644b8aaf77690c0b3ea2a006dc935f4d988be966594ec769cb32 not found: ID does not exist" Oct 30 01:48:58.775612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.775550 5035 scope.go:122] "RemoveContainer" containerID="6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42" Oct 30 01:48:58.775897 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:48:58.775879 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42\": container with ID starting with 6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42 not found: ID does not exist" containerID="6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42" Oct 30 01:48:58.775971 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.775920 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42"} err="failed to get container status \"6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42\": rpc error: code = NotFound desc = could not find container \"6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42\": container with ID starting with 6dc3743642e826b886cdc65afc0811a3e8a57fbc7500303534b4d8596584da42 not found: ID does not exist" Oct 30 01:48:58.787152 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.787125 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:58.802242 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.802203 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd"] Oct 30 01:48:58.815062 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.815026 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bc6b04f0-openshift-default-7c87448c5f-mtcbd"] Oct 30 01:48:58.862399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.862195 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-socket\") pod \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " Oct 30 01:48:58.862399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.862278 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-kube-api-access-s7hpq\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-kube-api-access-s7hpq\") pod \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " Oct 30 01:48:58.862399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.862332 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-certs\") pod \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " Oct 30 01:48:58.862789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.862410 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-envoy\") pod \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " Oct 30 01:48:58.862789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.862481 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-credential-socket\") pod \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " Oct 30 01:48:58.862789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.862562 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istiod-ca-cert\") pod \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " Oct 30 01:48:58.862789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.862639 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-token\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-token\") pod \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " Oct 30 01:48:58.862789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.862674 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-data\") pod \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " Oct 30 01:48:58.862789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.862713 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-podinfo\") pod \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\" (UID: \"40709a04-b185-4f5f-bdf3-294c6e3b4eed\") " Oct 30 01:48:58.864488 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.864122 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istiod-ca-cert" pod "40709a04-b185-4f5f-bdf3-294c6e3b4eed" (UID: "40709a04-b185-4f5f-bdf3-294c6e3b4eed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:58.864488 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.864196 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-credential-socket" pod "40709a04-b185-4f5f-bdf3-294c6e3b4eed" (UID: "40709a04-b185-4f5f-bdf3-294c6e3b4eed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.864488 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.864339 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-socket" pod "40709a04-b185-4f5f-bdf3-294c6e3b4eed" (UID: "40709a04-b185-4f5f-bdf3-294c6e3b4eed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.864488 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.864352 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-data" pod "40709a04-b185-4f5f-bdf3-294c6e3b4eed" (UID: "40709a04-b185-4f5f-bdf3-294c6e3b4eed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.864913 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.864519 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-certs" pod "40709a04-b185-4f5f-bdf3-294c6e3b4eed" (UID: "40709a04-b185-4f5f-bdf3-294c6e3b4eed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.866159 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.866122 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-podinfo" pod "40709a04-b185-4f5f-bdf3-294c6e3b4eed" (UID: "40709a04-b185-4f5f-bdf3-294c6e3b4eed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:48:58.866319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.866210 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-envoy" pod "40709a04-b185-4f5f-bdf3-294c6e3b4eed" (UID: "40709a04-b185-4f5f-bdf3-294c6e3b4eed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:58.866517 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.866490 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-token" pod "40709a04-b185-4f5f-bdf3-294c6e3b4eed" (UID: "40709a04-b185-4f5f-bdf3-294c6e3b4eed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:58.869505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.869472 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-kube-api-access-s7hpq" pod "40709a04-b185-4f5f-bdf3-294c6e3b4eed" (UID: "40709a04-b185-4f5f-bdf3-294c6e3b4eed"). InnerVolumeSpecName "kube-api-access-s7hpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:58.964422 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.964377 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.964422 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.964412 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.964422 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.964427 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.964745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.964439 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.964745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.964448 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.964745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.964460 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7hpq\" (UniqueName: \"kubernetes.io/projected/40709a04-b185-4f5f-bdf3-294c6e3b4eed-kube-api-access-s7hpq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.964745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.964469 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.964745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.964477 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:58.964745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:58.964485 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40709a04-b185-4f5f-bdf3-294c6e3b4eed-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:48:59.288117 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:59.288015 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:59.290002 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:59.289970 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:48:59.642966 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:59.642876 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" path="/var/lib/kubelet/pods/d46b78c8-d377-4f5e-a828-759559b2ed13/volumes" Oct 30 01:48:59.742781 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:59.742747 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" Oct 30 01:48:59.742961 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:59.742749 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg" event={"ID":"40709a04-b185-4f5f-bdf3-294c6e3b4eed","Type":"ContainerDied","Data":"757c273ee79b8c5f034c508af976cea86929bb47f7c057c5558ae87dd5f0b3c8"} Oct 30 01:48:59.742961 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:59.742887 5035 scope.go:122] "RemoveContainer" containerID="041d8e654dda9d24ee13ae205997ab7d249c12bd1acdcd13d9e391ff096f1cc2" Oct 30 01:48:59.799064 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:59.799025 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg"] Oct 30 01:48:59.811915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:59.811870 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7664a237-gw-openshift-default-6c4697dd5c-bw4tg"] Oct 30 01:48:59.861030 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:48:59.860961 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" podUID="6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed" containerName="payload-processing" containerID="cri-o://133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061" gracePeriod=25 Oct 30 01:49:00.011580 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.011147 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" podUID="2f78bad0-c46b-4f70-a56f-42173f06469c" containerName="payload-pre-processing" containerID="cri-o://1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d" gracePeriod=25 Oct 30 01:49:00.136302 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.136272 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:49:00.178548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.178499 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-plugins-config-volume\") pod \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\" (UID: \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\") " Oct 30 01:49:00.178708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.178658 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-kube-api-access-zrjhr\" (UniqueName: \"kubernetes.io/projected/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-kube-api-access-zrjhr\") pod \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\" (UID: \"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed\") " Oct 30 01:49:00.178990 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.178961 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-plugins-config-volume" pod "6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed" (UID: "6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:00.180912 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.180881 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-kube-api-access-zrjhr" pod "6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed" (UID: "6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed"). InnerVolumeSpecName "kube-api-access-zrjhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:00.279925 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.279819 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:00.279925 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.279849 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrjhr\" (UniqueName: \"kubernetes.io/projected/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed-kube-api-access-zrjhr\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:00.335124 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.335090 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:49:00.380680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.380459 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f78bad0-c46b-4f70-a56f-42173f06469c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f78bad0-c46b-4f70-a56f-42173f06469c-plugins-config-volume\") pod \"2f78bad0-c46b-4f70-a56f-42173f06469c\" (UID: \"2f78bad0-c46b-4f70-a56f-42173f06469c\") " Oct 30 01:49:00.380680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.380674 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f78bad0-c46b-4f70-a56f-42173f06469c-kube-api-access-x6fjp\" (UniqueName: \"kubernetes.io/projected/2f78bad0-c46b-4f70-a56f-42173f06469c-kube-api-access-x6fjp\") pod \"2f78bad0-c46b-4f70-a56f-42173f06469c\" (UID: \"2f78bad0-c46b-4f70-a56f-42173f06469c\") " Oct 30 01:49:00.380995 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.380964 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f78bad0-c46b-4f70-a56f-42173f06469c-plugins-config-volume" pod "2f78bad0-c46b-4f70-a56f-42173f06469c" (UID: "2f78bad0-c46b-4f70-a56f-42173f06469c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:00.381168 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.381136 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f78bad0-c46b-4f70-a56f-42173f06469c-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:00.383556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.383497 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f78bad0-c46b-4f70-a56f-42173f06469c-kube-api-access-x6fjp" pod "2f78bad0-c46b-4f70-a56f-42173f06469c" (UID: "2f78bad0-c46b-4f70-a56f-42173f06469c"). InnerVolumeSpecName "kube-api-access-x6fjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:00.466183 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466145 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6"] Oct 30 01:49:00.466639 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466624 5035 cpu_manager.go:414] "containerMap: removing container" podUID="2f78bad0-c46b-4f70-a56f-42173f06469c" containerName="payload-pre-processing" Oct 30 01:49:00.466702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466640 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f78bad0-c46b-4f70-a56f-42173f06469c" containerName="payload-pre-processing" Oct 30 01:49:00.466702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466662 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed" containerName="payload-processing" Oct 30 01:49:00.466702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466667 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed" containerName="payload-processing" Oct 30 01:49:00.466702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466685 5035 cpu_manager.go:414] "containerMap: removing container" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" Oct 30 01:49:00.466702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466690 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" Oct 30 01:49:00.466702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466699 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" Oct 30 01:49:00.466702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466703 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466711 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466715 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466735 5035 cpu_manager.go:414] "containerMap: removing container" podUID="968430eb-9bf4-4f33-b8fc-0c8cfe34dafc" containerName="payload-pre-processing" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466740 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="968430eb-9bf4-4f33-b8fc-0c8cfe34dafc" containerName="payload-pre-processing" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466746 5035 cpu_manager.go:414] "containerMap: removing container" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466750 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466817 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466825 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466830 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46b78c8-d377-4f5e-a828-759559b2ed13" containerName="istio-proxy" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466838 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="968430eb-9bf4-4f33-b8fc-0c8cfe34dafc" containerName="payload-pre-processing" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466844 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" containerName="istio-proxy" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466850 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f78bad0-c46b-4f70-a56f-42173f06469c" containerName="payload-pre-processing" Oct 30 01:49:00.466907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.466857 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed" containerName="payload-processing" Oct 30 01:49:00.470407 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.470381 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:00.481952 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.481917 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6fjp\" (UniqueName: \"kubernetes.io/projected/2f78bad0-c46b-4f70-a56f-42173f06469c-kube-api-access-x6fjp\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:00.486702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.486651 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-78d5ff\"" Oct 30 01:49:00.490379 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.490346 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6"] Oct 30 01:49:00.525562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.522949 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6"] Oct 30 01:49:00.535345 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.535116 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:00.546692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.546377 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6"] Oct 30 01:49:00.578145 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.578108 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-78d5ff-dockercfg-hwqnw\"" Oct 30 01:49:00.583050 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.582994 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6\" (UID: \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:00.583050 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.583054 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e5e32e-4e1a-4585-967c-d98ad0617ec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6\" (UID: \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:00.583324 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.583119 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4ww\" (UniqueName: \"kubernetes.io/projected/00e5e32e-4e1a-4585-967c-d98ad0617ec9-kube-api-access-km4ww\") pod \"payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6\" (UID: \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:00.583324 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.583164 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrxq5\" (UniqueName: \"kubernetes.io/projected/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-kube-api-access-wrxq5\") pod \"payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6\" (UID: \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:00.684802 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.684755 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e5e32e-4e1a-4585-967c-d98ad0617ec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6\" (UID: \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:00.684974 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.684840 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4ww\" (UniqueName: \"kubernetes.io/projected/00e5e32e-4e1a-4585-967c-d98ad0617ec9-kube-api-access-km4ww\") pod \"payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6\" (UID: \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:00.684974 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.684873 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrxq5\" (UniqueName: \"kubernetes.io/projected/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-kube-api-access-wrxq5\") pod \"payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6\" (UID: \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:00.685098 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.685008 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6\" (UID: \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:00.685860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.685833 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6\" (UID: \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:00.685942 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.685833 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e5e32e-4e1a-4585-967c-d98ad0617ec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6\" (UID: \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:00.709520 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.709482 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrxq5\" (UniqueName: \"kubernetes.io/projected/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-kube-api-access-wrxq5\") pod \"payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6\" (UID: \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:00.709687 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.709494 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4ww\" (UniqueName: \"kubernetes.io/projected/00e5e32e-4e1a-4585-967c-d98ad0617ec9-kube-api-access-km4ww\") pod \"payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6\" (UID: \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:00.750874 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.750835 5035 generic.go:356] "Generic (PLEG): container finished" podID="6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed" containerID="133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061" exitCode=0 Oct 30 01:49:00.751084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.750889 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" Oct 30 01:49:00.751084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.750912 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" event={"ID":"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed","Type":"ContainerDied","Data":"133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061"} Oct 30 01:49:00.751084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.750983 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s" event={"ID":"6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed","Type":"ContainerDied","Data":"f2fd142e41c7003bc481b607b4769f1d0ea4d2344b0cd89de651c28c9ec4235c"} Oct 30 01:49:00.751084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.751018 5035 scope.go:122] "RemoveContainer" containerID="133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061" Oct 30 01:49:00.752643 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.752616 5035 generic.go:356] "Generic (PLEG): container finished" podID="2f78bad0-c46b-4f70-a56f-42173f06469c" containerID="1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d" exitCode=0 Oct 30 01:49:00.752792 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.752653 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" event={"ID":"2f78bad0-c46b-4f70-a56f-42173f06469c","Type":"ContainerDied","Data":"1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d"} Oct 30 01:49:00.752792 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.752680 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" Oct 30 01:49:00.752792 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.752719 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74" event={"ID":"2f78bad0-c46b-4f70-a56f-42173f06469c","Type":"ContainerDied","Data":"5f0a36086998d4c5adb5ec9ee8f2eda7c517af9032f6721683681626e5871e6c"} Oct 30 01:49:00.763605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.763581 5035 scope.go:122] "RemoveContainer" containerID="133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061" Oct 30 01:49:00.763976 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:00.763949 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061\": container with ID starting with 133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061 not found: ID does not exist" containerID="133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061" Oct 30 01:49:00.764083 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.763987 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061"} err="failed to get container status \"133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061\": rpc error: code = NotFound desc = could not find container \"133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061\": container with ID starting with 133ef390741d278ae5ae147a8f0998d7c1f6a3fa2508c7ed7be090c33dd23061 not found: ID does not exist" Oct 30 01:49:00.764083 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.764013 5035 scope.go:122] "RemoveContainer" containerID="1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d" Oct 30 01:49:00.774146 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.774125 5035 scope.go:122] "RemoveContainer" containerID="1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d" Oct 30 01:49:00.774473 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:00.774454 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d\": container with ID starting with 1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d not found: ID does not exist" containerID="1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d" Oct 30 01:49:00.774520 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.774481 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d"} err="failed to get container status \"1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d\": rpc error: code = NotFound desc = could not find container \"1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d\": container with ID starting with 1179320555c10bae754196ef1b73543eab84805d7ff6d1f5fedc319a1248710d not found: ID does not exist" Oct 30 01:49:00.785123 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.785090 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:00.805366 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.805327 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s"] Oct 30 01:49:00.809569 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.809536 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-61b6469a-d856545d4-2z75s"] Oct 30 01:49:00.846726 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.845814 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74"] Oct 30 01:49:00.849265 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.849235 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-61b6469a-6fc6d565dc-p6v74"] Oct 30 01:49:00.873103 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:00.873068 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:01.101665 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.101625 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6"] Oct 30 01:49:01.179499 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.179472 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6"] Oct 30 01:49:01.181214 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:49:01.181179 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5d3bb8_7e46_48c0_bb2c_ffd5e7337636.slice/crio-d665872ad1da5b6995fe1f3cc908ddc2d4f43a9500d97f4d01d4ae5aac74edab WatchSource:0}: Error finding container d665872ad1da5b6995fe1f3cc908ddc2d4f43a9500d97f4d01d4ae5aac74edab: Status 404 returned error can't find the container with id d665872ad1da5b6995fe1f3cc908ddc2d4f43a9500d97f4d01d4ae5aac74edab Oct 30 01:49:01.652961 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.652924 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f78bad0-c46b-4f70-a56f-42173f06469c" path="/var/lib/kubelet/pods/2f78bad0-c46b-4f70-a56f-42173f06469c/volumes" Oct 30 01:49:01.653403 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.653388 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40709a04-b185-4f5f-bdf3-294c6e3b4eed" path="/var/lib/kubelet/pods/40709a04-b185-4f5f-bdf3-294c6e3b4eed/volumes" Oct 30 01:49:01.653767 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.653754 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed" path="/var/lib/kubelet/pods/6758ea8e-24ef-4f79-9f88-e4b74f4cd8ed/volumes" Oct 30 01:49:01.761990 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.761943 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" event={"ID":"00e5e32e-4e1a-4585-967c-d98ad0617ec9","Type":"ContainerStarted","Data":"dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548"} Oct 30 01:49:01.762175 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.762007 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" event={"ID":"00e5e32e-4e1a-4585-967c-d98ad0617ec9","Type":"ContainerStarted","Data":"4a2ced410e66433dd09dde075228b6d9ff7cd24a5790b5922d37cfd590bc1381"} Oct 30 01:49:01.762175 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.762156 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:01.764636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.764604 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" event={"ID":"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636","Type":"ContainerStarted","Data":"ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d"} Oct 30 01:49:01.764805 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.764657 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" event={"ID":"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636","Type":"ContainerStarted","Data":"d665872ad1da5b6995fe1f3cc908ddc2d4f43a9500d97f4d01d4ae5aac74edab"} Oct 30 01:49:01.764805 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.764746 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:01.806731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.806677 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" podStartSLOduration=1.806662971 podStartE2EDuration="1.806662971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:01.804892624 +0000 UTC m=+1220.307824844" watchObservedRunningTime="2026-10-30 01:49:01.806662971 +0000 UTC m=+1220.309595191" Oct 30 01:49:01.851079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.851007 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" podStartSLOduration=1.850985132 podStartE2EDuration="1.850985132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:01.847889802 +0000 UTC m=+1220.350822023" watchObservedRunningTime="2026-10-30 01:49:01.850985132 +0000 UTC m=+1220.353917354" Oct 30 01:49:01.928704 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:01.928599 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" podUID="27420bb1-897c-4ce7-933f-c95780592774" containerName="payload-pre-processing" containerID="cri-o://2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423" gracePeriod=25 Oct 30 01:49:02.222827 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.222792 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:49:02.307839 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.307792 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27420bb1-897c-4ce7-933f-c95780592774-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27420bb1-897c-4ce7-933f-c95780592774-plugins-config-volume\") pod \"27420bb1-897c-4ce7-933f-c95780592774\" (UID: \"27420bb1-897c-4ce7-933f-c95780592774\") " Oct 30 01:49:02.308011 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.307985 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27420bb1-897c-4ce7-933f-c95780592774-kube-api-access-ccj7c\" (UniqueName: \"kubernetes.io/projected/27420bb1-897c-4ce7-933f-c95780592774-kube-api-access-ccj7c\") pod \"27420bb1-897c-4ce7-933f-c95780592774\" (UID: \"27420bb1-897c-4ce7-933f-c95780592774\") " Oct 30 01:49:02.308202 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.308176 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27420bb1-897c-4ce7-933f-c95780592774-plugins-config-volume" pod "27420bb1-897c-4ce7-933f-c95780592774" (UID: "27420bb1-897c-4ce7-933f-c95780592774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:02.308470 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.308437 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27420bb1-897c-4ce7-933f-c95780592774-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:02.310144 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.310115 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27420bb1-897c-4ce7-933f-c95780592774-kube-api-access-ccj7c" pod "27420bb1-897c-4ce7-933f-c95780592774" (UID: "27420bb1-897c-4ce7-933f-c95780592774"). InnerVolumeSpecName "kube-api-access-ccj7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:02.348962 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.348923 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6"] Oct 30 01:49:02.349426 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.349412 5035 cpu_manager.go:414] "containerMap: removing container" podUID="27420bb1-897c-4ce7-933f-c95780592774" containerName="payload-pre-processing" Oct 30 01:49:02.349465 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.349427 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27420bb1-897c-4ce7-933f-c95780592774" containerName="payload-pre-processing" Oct 30 01:49:02.349513 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.349504 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="27420bb1-897c-4ce7-933f-c95780592774" containerName="payload-pre-processing" Oct 30 01:49:02.352807 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.352787 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:02.371374 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.371340 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-719029\"" Oct 30 01:49:02.371936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.371899 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-719029-dockercfg-wltvm\"" Oct 30 01:49:02.372989 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.372967 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6"] Oct 30 01:49:02.409469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.409426 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f27a327-a17e-45e6-80fe-88d732edb488-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6\" (UID: \"6f27a327-a17e-45e6-80fe-88d732edb488\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:02.409719 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.409504 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ht4f\" (UniqueName: \"kubernetes.io/projected/6f27a327-a17e-45e6-80fe-88d732edb488-kube-api-access-4ht4f\") pod \"payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6\" (UID: \"6f27a327-a17e-45e6-80fe-88d732edb488\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:02.409848 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.409824 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccj7c\" (UniqueName: \"kubernetes.io/projected/27420bb1-897c-4ce7-933f-c95780592774-kube-api-access-ccj7c\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:02.410990 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.410958 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz"] Oct 30 01:49:02.448831 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.415625 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:02.448831 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.445101 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz"] Oct 30 01:49:02.510404 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.510360 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f27a327-a17e-45e6-80fe-88d732edb488-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6\" (UID: \"6f27a327-a17e-45e6-80fe-88d732edb488\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:02.510637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.510432 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p72l7\" (UniqueName: \"kubernetes.io/projected/13653c11-c510-447b-90fb-2d959bee34d0-kube-api-access-p72l7\") pod \"payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz\" (UID: \"13653c11-c510-447b-90fb-2d959bee34d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:02.510637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.510477 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ht4f\" (UniqueName: \"kubernetes.io/projected/6f27a327-a17e-45e6-80fe-88d732edb488-kube-api-access-4ht4f\") pod \"payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6\" (UID: \"6f27a327-a17e-45e6-80fe-88d732edb488\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:02.510637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.510621 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13653c11-c510-447b-90fb-2d959bee34d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz\" (UID: \"13653c11-c510-447b-90fb-2d959bee34d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:02.511136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.511114 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f27a327-a17e-45e6-80fe-88d732edb488-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6\" (UID: \"6f27a327-a17e-45e6-80fe-88d732edb488\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:02.534935 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.534899 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ht4f\" (UniqueName: \"kubernetes.io/projected/6f27a327-a17e-45e6-80fe-88d732edb488-kube-api-access-4ht4f\") pod \"payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6\" (UID: \"6f27a327-a17e-45e6-80fe-88d732edb488\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:02.611952 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.611902 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p72l7\" (UniqueName: \"kubernetes.io/projected/13653c11-c510-447b-90fb-2d959bee34d0-kube-api-access-p72l7\") pod \"payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz\" (UID: \"13653c11-c510-447b-90fb-2d959bee34d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:02.612143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.611990 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13653c11-c510-447b-90fb-2d959bee34d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz\" (UID: \"13653c11-c510-447b-90fb-2d959bee34d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:02.612797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.612774 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13653c11-c510-447b-90fb-2d959bee34d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz\" (UID: \"13653c11-c510-447b-90fb-2d959bee34d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:02.636330 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.636292 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p72l7\" (UniqueName: \"kubernetes.io/projected/13653c11-c510-447b-90fb-2d959bee34d0-kube-api-access-p72l7\") pod \"payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz\" (UID: \"13653c11-c510-447b-90fb-2d959bee34d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:02.666745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.666702 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:02.756026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.755983 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:02.773543 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.773502 5035 generic.go:356] "Generic (PLEG): container finished" podID="27420bb1-897c-4ce7-933f-c95780592774" containerID="2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423" exitCode=0 Oct 30 01:49:02.773677 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.773579 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" Oct 30 01:49:02.773677 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.773639 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" event={"ID":"27420bb1-897c-4ce7-933f-c95780592774","Type":"ContainerDied","Data":"2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423"} Oct 30 01:49:02.773756 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.773678 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v" event={"ID":"27420bb1-897c-4ce7-933f-c95780592774","Type":"ContainerDied","Data":"a44a12cf353b6974495e2ce6fe2cf590d65f5cb82e90ed38305dc09a7ed8145a"} Oct 30 01:49:02.773756 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.773702 5035 scope.go:122] "RemoveContainer" containerID="2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423" Oct 30 01:49:02.786514 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.786471 5035 scope.go:122] "RemoveContainer" containerID="2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423" Oct 30 01:49:02.787404 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:02.787375 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423\": container with ID starting with 2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423 not found: ID does not exist" containerID="2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423" Oct 30 01:49:02.788502 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.787585 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423"} err="failed to get container status \"2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423\": rpc error: code = NotFound desc = could not find container \"2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423\": container with ID starting with 2316400c33100fb72cc0f65ce5ed950b202e94ff069a64b3777e879749593423 not found: ID does not exist" Oct 30 01:49:02.834365 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.834331 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v"] Oct 30 01:49:02.836903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.836871 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-14b53b75-546f8c66ff-zb98v"] Oct 30 01:49:02.966797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:02.966763 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6"] Oct 30 01:49:02.970142 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:49:02.970075 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f27a327_a17e_45e6_80fe_88d732edb488.slice/crio-0a440c8726ff3481e4de8e0385eed75934bee1940f0a09af018024645e317f01 WatchSource:0}: Error finding container 0a440c8726ff3481e4de8e0385eed75934bee1940f0a09af018024645e317f01: Status 404 returned error can't find the container with id 0a440c8726ff3481e4de8e0385eed75934bee1940f0a09af018024645e317f01 Oct 30 01:49:03.099263 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:03.099202 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz"] Oct 30 01:49:03.642306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:03.642270 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27420bb1-897c-4ce7-933f-c95780592774" path="/var/lib/kubelet/pods/27420bb1-897c-4ce7-933f-c95780592774/volumes" Oct 30 01:49:03.781593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:03.781550 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" event={"ID":"13653c11-c510-447b-90fb-2d959bee34d0","Type":"ContainerStarted","Data":"e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787"} Oct 30 01:49:03.782021 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:03.781621 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:03.782021 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:03.781647 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" event={"ID":"13653c11-c510-447b-90fb-2d959bee34d0","Type":"ContainerStarted","Data":"091e209390eb7448e482d8e89cb33fb4cd54a53989917863ab47c6c892c00ceb"} Oct 30 01:49:03.784263 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:03.784235 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" event={"ID":"6f27a327-a17e-45e6-80fe-88d732edb488","Type":"ContainerStarted","Data":"16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7"} Oct 30 01:49:03.784389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:03.784283 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" event={"ID":"6f27a327-a17e-45e6-80fe-88d732edb488","Type":"ContainerStarted","Data":"0a440c8726ff3481e4de8e0385eed75934bee1940f0a09af018024645e317f01"} Oct 30 01:49:03.784389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:03.784304 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:03.827572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:03.827200 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" podStartSLOduration=1.827180034 podStartE2EDuration="1.827180034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:03.824828904 +0000 UTC m=+1222.327761124" watchObservedRunningTime="2026-10-30 01:49:03.827180034 +0000 UTC m=+1222.330112255" Oct 30 01:49:07.695408 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.695321 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" podStartSLOduration=5.695297402 podStartE2EDuration="5.695297402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:03.873418973 +0000 UTC m=+1222.376351193" watchObservedRunningTime="2026-10-30 01:49:07.695297402 +0000 UTC m=+1226.198229624" Oct 30 01:49:07.695939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.695860 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl"] Oct 30 01:49:07.703039 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.703008 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl"] Oct 30 01:49:07.703181 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.703146 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:07.719555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.719463 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c40a89\"" Oct 30 01:49:07.736899 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.736861 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c40a89-dockercfg-xv9fv\"" Oct 30 01:49:07.803552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.800256 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r89z\" (UniqueName: \"kubernetes.io/projected/d07d3256-13c9-4f87-a5b1-7d50bbaed683-kube-api-access-6r89z\") pod \"payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl\" (UID: \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:07.803552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.800391 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d07d3256-13c9-4f87-a5b1-7d50bbaed683-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl\" (UID: \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:07.805107 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.805071 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6"] Oct 30 01:49:07.812128 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.812095 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6"] Oct 30 01:49:07.812311 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.812202 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:07.901767 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.901725 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhlq\" (UniqueName: \"kubernetes.io/projected/605e1a69-357a-4cba-8a4a-b88c8bb5f146-kube-api-access-rwhlq\") pod \"payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6\" (UID: \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:07.901767 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.901772 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d07d3256-13c9-4f87-a5b1-7d50bbaed683-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl\" (UID: \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:07.901989 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.901953 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605e1a69-357a-4cba-8a4a-b88c8bb5f146-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6\" (UID: \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:07.902041 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.902023 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r89z\" (UniqueName: \"kubernetes.io/projected/d07d3256-13c9-4f87-a5b1-7d50bbaed683-kube-api-access-6r89z\") pod \"payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl\" (UID: \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:07.902645 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.902626 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d07d3256-13c9-4f87-a5b1-7d50bbaed683-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl\" (UID: \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:07.926601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:07.926562 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r89z\" (UniqueName: \"kubernetes.io/projected/d07d3256-13c9-4f87-a5b1-7d50bbaed683-kube-api-access-6r89z\") pod \"payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl\" (UID: \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:08.002855 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.002800 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605e1a69-357a-4cba-8a4a-b88c8bb5f146-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6\" (UID: \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:08.003059 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.002909 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhlq\" (UniqueName: \"kubernetes.io/projected/605e1a69-357a-4cba-8a4a-b88c8bb5f146-kube-api-access-rwhlq\") pod \"payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6\" (UID: \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:08.003626 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.003603 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605e1a69-357a-4cba-8a4a-b88c8bb5f146-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6\" (UID: \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:08.019941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.019886 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:08.027548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.027501 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhlq\" (UniqueName: \"kubernetes.io/projected/605e1a69-357a-4cba-8a4a-b88c8bb5f146-kube-api-access-rwhlq\") pod \"payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6\" (UID: \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:08.131627 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.131518 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:08.352502 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:49:08.352465 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07d3256_13c9_4f87_a5b1_7d50bbaed683.slice/crio-8a9e107557a64ecd005601e974bab26686827d4aa3c1a924811665f5cadf959b WatchSource:0}: Error finding container 8a9e107557a64ecd005601e974bab26686827d4aa3c1a924811665f5cadf959b: Status 404 returned error can't find the container with id 8a9e107557a64ecd005601e974bab26686827d4aa3c1a924811665f5cadf959b Oct 30 01:49:08.353274 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.353240 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl"] Oct 30 01:49:08.496363 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.496318 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6"] Oct 30 01:49:08.498624 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:49:08.498596 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605e1a69_357a_4cba_8a4a_b88c8bb5f146.slice/crio-a1fd566d70b4253fc06447e78ba582f80a33b96fb8854748466aa0bacc798cab WatchSource:0}: Error finding container a1fd566d70b4253fc06447e78ba582f80a33b96fb8854748466aa0bacc798cab: Status 404 returned error can't find the container with id a1fd566d70b4253fc06447e78ba582f80a33b96fb8854748466aa0bacc798cab Oct 30 01:49:08.693227 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.693129 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj"] Oct 30 01:49:08.726851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.726815 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq"] Oct 30 01:49:08.822108 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.822069 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" event={"ID":"605e1a69-357a-4cba-8a4a-b88c8bb5f146","Type":"ContainerStarted","Data":"8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c"} Oct 30 01:49:08.822284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.822137 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" event={"ID":"605e1a69-357a-4cba-8a4a-b88c8bb5f146","Type":"ContainerStarted","Data":"a1fd566d70b4253fc06447e78ba582f80a33b96fb8854748466aa0bacc798cab"} Oct 30 01:49:08.823174 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.823151 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:08.826444 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.826408 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" event={"ID":"d07d3256-13c9-4f87-a5b1-7d50bbaed683","Type":"ContainerStarted","Data":"49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689"} Oct 30 01:49:08.826655 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.826468 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" event={"ID":"d07d3256-13c9-4f87-a5b1-7d50bbaed683","Type":"ContainerStarted","Data":"8a9e107557a64ecd005601e974bab26686827d4aa3c1a924811665f5cadf959b"} Oct 30 01:49:08.826655 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.826579 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:08.865787 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.865732 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" podStartSLOduration=1.8657164320000001 podStartE2EDuration="1.865716432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:08.865287052 +0000 UTC m=+1227.368219272" watchObservedRunningTime="2026-10-30 01:49:08.865716432 +0000 UTC m=+1227.368648651" Oct 30 01:49:08.922886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:08.922821 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" podStartSLOduration=1.92280182 podStartE2EDuration="1.92280182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:08.921701093 +0000 UTC m=+1227.424633316" watchObservedRunningTime="2026-10-30 01:49:08.92280182 +0000 UTC m=+1227.425734045" Oct 30 01:49:12.777600 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:12.777493 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 01:49:12.777981 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:12.777677 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 01:49:13.694063 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:13.694010 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" podUID="aa99545f-be08-4953-8537-54935ebcceb3" containerName="payload-processing" containerID="cri-o://1d0749bd1c651886ce36fc66afb4c4f297ec765c03c1779716f4b98da8da0648" gracePeriod=25 Oct 30 01:49:13.727634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:13.727583 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" podUID="604332d8-bef6-4ba2-85d9-65fa46b15d79" containerName="payload-pre-processing" containerID="cri-o://9a9a9fafc5ea754397a494ec47fd1acfd0165984ceb890e4c63131fe94f4a929" gracePeriod=25 Oct 30 01:49:13.868645 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:13.868607 5035 generic.go:356] "Generic (PLEG): container finished" podID="604332d8-bef6-4ba2-85d9-65fa46b15d79" containerID="9a9a9fafc5ea754397a494ec47fd1acfd0165984ceb890e4c63131fe94f4a929" exitCode=0 Oct 30 01:49:13.869067 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:13.868704 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" event={"ID":"604332d8-bef6-4ba2-85d9-65fa46b15d79","Type":"ContainerDied","Data":"9a9a9fafc5ea754397a494ec47fd1acfd0165984ceb890e4c63131fe94f4a929"} Oct 30 01:49:13.871702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:13.871663 5035 generic.go:356] "Generic (PLEG): container finished" podID="aa99545f-be08-4953-8537-54935ebcceb3" containerID="1d0749bd1c651886ce36fc66afb4c4f297ec765c03c1779716f4b98da8da0648" exitCode=0 Oct 30 01:49:13.871852 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:13.871733 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" event={"ID":"aa99545f-be08-4953-8537-54935ebcceb3","Type":"ContainerDied","Data":"1d0749bd1c651886ce36fc66afb4c4f297ec765c03c1779716f4b98da8da0648"} Oct 30 01:49:14.032786 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.032758 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:49:14.036682 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.036653 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:49:14.070886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.070845 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604332d8-bef6-4ba2-85d9-65fa46b15d79-kube-api-access-2mptf\" (UniqueName: \"kubernetes.io/projected/604332d8-bef6-4ba2-85d9-65fa46b15d79-kube-api-access-2mptf\") pod \"604332d8-bef6-4ba2-85d9-65fa46b15d79\" (UID: \"604332d8-bef6-4ba2-85d9-65fa46b15d79\") " Oct 30 01:49:14.073051 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.073015 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604332d8-bef6-4ba2-85d9-65fa46b15d79-kube-api-access-2mptf" pod "604332d8-bef6-4ba2-85d9-65fa46b15d79" (UID: "604332d8-bef6-4ba2-85d9-65fa46b15d79"). InnerVolumeSpecName "kube-api-access-2mptf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:14.171643 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.171601 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604332d8-bef6-4ba2-85d9-65fa46b15d79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604332d8-bef6-4ba2-85d9-65fa46b15d79-plugins-config-volume\") pod \"604332d8-bef6-4ba2-85d9-65fa46b15d79\" (UID: \"604332d8-bef6-4ba2-85d9-65fa46b15d79\") " Oct 30 01:49:14.171850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.171753 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa99545f-be08-4953-8537-54935ebcceb3-kube-api-access-l2dv8\" (UniqueName: \"kubernetes.io/projected/aa99545f-be08-4953-8537-54935ebcceb3-kube-api-access-l2dv8\") pod \"aa99545f-be08-4953-8537-54935ebcceb3\" (UID: \"aa99545f-be08-4953-8537-54935ebcceb3\") " Oct 30 01:49:14.171850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.171784 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa99545f-be08-4953-8537-54935ebcceb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa99545f-be08-4953-8537-54935ebcceb3-plugins-config-volume\") pod \"aa99545f-be08-4953-8537-54935ebcceb3\" (UID: \"aa99545f-be08-4953-8537-54935ebcceb3\") " Oct 30 01:49:14.172084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.172056 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604332d8-bef6-4ba2-85d9-65fa46b15d79-plugins-config-volume" pod "604332d8-bef6-4ba2-85d9-65fa46b15d79" (UID: "604332d8-bef6-4ba2-85d9-65fa46b15d79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:14.172139 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.172062 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mptf\" (UniqueName: \"kubernetes.io/projected/604332d8-bef6-4ba2-85d9-65fa46b15d79-kube-api-access-2mptf\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:14.172329 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.172302 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa99545f-be08-4953-8537-54935ebcceb3-plugins-config-volume" pod "aa99545f-be08-4953-8537-54935ebcceb3" (UID: "aa99545f-be08-4953-8537-54935ebcceb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:14.174177 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.174149 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa99545f-be08-4953-8537-54935ebcceb3-kube-api-access-l2dv8" pod "aa99545f-be08-4953-8537-54935ebcceb3" (UID: "aa99545f-be08-4953-8537-54935ebcceb3"). InnerVolumeSpecName "kube-api-access-l2dv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:14.273084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.273046 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2dv8\" (UniqueName: \"kubernetes.io/projected/aa99545f-be08-4953-8537-54935ebcceb3-kube-api-access-l2dv8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:14.273084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.273079 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa99545f-be08-4953-8537-54935ebcceb3-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:14.273084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.273089 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604332d8-bef6-4ba2-85d9-65fa46b15d79-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:14.793363 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.793333 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 01:49:14.793567 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.793405 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 01:49:14.882290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.882249 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" event={"ID":"604332d8-bef6-4ba2-85d9-65fa46b15d79","Type":"ContainerDied","Data":"ab1673224dfffa80e03311a3121601beef7f53c0563787d0dc54af8c7786b58b"} Oct 30 01:49:14.882816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.882331 5035 scope.go:122] "RemoveContainer" containerID="9a9a9fafc5ea754397a494ec47fd1acfd0165984ceb890e4c63131fe94f4a929" Oct 30 01:49:14.882923 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.882893 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq" Oct 30 01:49:14.884631 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.884602 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" event={"ID":"aa99545f-be08-4953-8537-54935ebcceb3","Type":"ContainerDied","Data":"28fc09abf76a359fd394605037ff996e94595ceaaf54044f0998e3c3d0f06bb1"} Oct 30 01:49:14.884751 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.884651 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj" Oct 30 01:49:14.908403 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.908368 5035 scope.go:122] "RemoveContainer" containerID="1d0749bd1c651886ce36fc66afb4c4f297ec765c03c1779716f4b98da8da0648" Oct 30 01:49:14.944496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.944461 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq"] Oct 30 01:49:14.950485 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:14.950453 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6c3505dc-79bf4bd659-d99vq"] Oct 30 01:49:15.013631 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:15.013479 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj"] Oct 30 01:49:15.013631 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:15.013552 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6c3505dc-68c5ff6c98-qgtfj"] Oct 30 01:49:15.643777 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:15.643729 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604332d8-bef6-4ba2-85d9-65fa46b15d79" path="/var/lib/kubelet/pods/604332d8-bef6-4ba2-85d9-65fa46b15d79/volumes" Oct 30 01:49:15.644182 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:15.644163 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa99545f-be08-4953-8537-54935ebcceb3" path="/var/lib/kubelet/pods/aa99545f-be08-4953-8537-54935ebcceb3/volumes" Oct 30 01:49:16.900904 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:16.900864 5035 generic.go:356] "Generic (PLEG): container finished" podID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerID="f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded" exitCode=0 Oct 30 01:49:16.900904 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:16.900907 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" event={"ID":"b5a7ded3-59ff-4e1c-bb3a-00956659d646","Type":"ContainerDied","Data":"f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded"} Oct 30 01:49:16.901554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:16.901398 5035 scope.go:122] "RemoveContainer" containerID="f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded" Oct 30 01:49:16.902813 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:16.902780 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:16.902914 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:16.902858 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:16.902914 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:16.902894 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:17.629555 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:17.629502 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:49:17.909592 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:17.909489 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" event={"ID":"b5a7ded3-59ff-4e1c-bb3a-00956659d646","Type":"ContainerStarted","Data":"ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab"} Oct 30 01:49:18.125049 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.125012 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp"] Oct 30 01:49:18.125273 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.125241 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" podUID="5438ad60-7f7b-4778-b56a-37a06740e8ee" containerName="maas-api" containerID="cri-o://b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78" gracePeriod=30 Oct 30 01:49:18.128118 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.128084 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:49:18.128396 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.128368 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:18.128745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.128445 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 01:49:18.449892 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.449862 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:49:18.516868 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.516765 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls\") pod \"5438ad60-7f7b-4778-b56a-37a06740e8ee\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " Oct 30 01:49:18.516868 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.516817 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5438ad60-7f7b-4778-b56a-37a06740e8ee-kube-api-access-8h82j\" (UniqueName: \"kubernetes.io/projected/5438ad60-7f7b-4778-b56a-37a06740e8ee-kube-api-access-8h82j\") pod \"5438ad60-7f7b-4778-b56a-37a06740e8ee\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " Oct 30 01:49:18.516868 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.516857 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-metrics-certs\") pod \"5438ad60-7f7b-4778-b56a-37a06740e8ee\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " Oct 30 01:49:18.517167 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.516912 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5438ad60-7f7b-4778-b56a-37a06740e8ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5438ad60-7f7b-4778-b56a-37a06740e8ee-openshift-service-ca\") pod \"5438ad60-7f7b-4778-b56a-37a06740e8ee\" (UID: \"5438ad60-7f7b-4778-b56a-37a06740e8ee\") " Oct 30 01:49:18.517979 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.517946 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5438ad60-7f7b-4778-b56a-37a06740e8ee-openshift-service-ca" pod "5438ad60-7f7b-4778-b56a-37a06740e8ee" (UID: "5438ad60-7f7b-4778-b56a-37a06740e8ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:18.520438 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.520403 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-metrics-certs" pod "5438ad60-7f7b-4778-b56a-37a06740e8ee" (UID: "5438ad60-7f7b-4778-b56a-37a06740e8ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:49:18.548228 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.545250 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls" pod "5438ad60-7f7b-4778-b56a-37a06740e8ee" (UID: "5438ad60-7f7b-4778-b56a-37a06740e8ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:49:18.549086 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.549044 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5438ad60-7f7b-4778-b56a-37a06740e8ee-kube-api-access-8h82j" pod "5438ad60-7f7b-4778-b56a-37a06740e8ee" (UID: "5438ad60-7f7b-4778-b56a-37a06740e8ee"). InnerVolumeSpecName "kube-api-access-8h82j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:18.618479 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.618442 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:18.618479 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.618476 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h82j\" (UniqueName: \"kubernetes.io/projected/5438ad60-7f7b-4778-b56a-37a06740e8ee-kube-api-access-8h82j\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:18.618479 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.618486 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5438ad60-7f7b-4778-b56a-37a06740e8ee-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:18.618742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.618495 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5438ad60-7f7b-4778-b56a-37a06740e8ee-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:18.917316 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.917220 5035 generic.go:356] "Generic (PLEG): container finished" podID="5438ad60-7f7b-4778-b56a-37a06740e8ee" containerID="b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78" exitCode=0 Oct 30 01:49:18.917316 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.917266 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" Oct 30 01:49:18.917849 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.917308 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" event={"ID":"5438ad60-7f7b-4778-b56a-37a06740e8ee","Type":"ContainerDied","Data":"b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78"} Oct 30 01:49:18.917849 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.917361 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp" event={"ID":"5438ad60-7f7b-4778-b56a-37a06740e8ee","Type":"ContainerDied","Data":"a8a7a2ebb9744919a34c0be2cafa7a5f89be4dcb4451a970b7c359011b55e87a"} Oct 30 01:49:18.917849 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.917389 5035 scope.go:122] "RemoveContainer" containerID="b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78" Oct 30 01:49:18.928203 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.928182 5035 scope.go:122] "RemoveContainer" containerID="b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78" Oct 30 01:49:18.928484 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:18.928458 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78\": container with ID starting with b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78 not found: ID does not exist" containerID="b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78" Oct 30 01:49:18.928580 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:18.928493 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78"} err="failed to get container status \"b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78\": rpc error: code = NotFound desc = could not find container \"b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78\": container with ID starting with b6e1666d5150673d25adf69abc0ac3e1bddd242fea188faddc8cbcd30cec3d78 not found: ID does not exist" Oct 30 01:49:19.003663 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:19.003611 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp"] Oct 30 01:49:19.006028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:19.005995 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6c3505dc-76566865b8-qpbzp"] Oct 30 01:49:19.034407 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:19.034366 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd"] Oct 30 01:49:19.034679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:19.034642 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" containerID="cri-o://fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b" gracePeriod=30 Oct 30 01:49:19.128698 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:19.128659 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:19.128893 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:19.128715 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 01:49:19.643649 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:19.643611 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5438ad60-7f7b-4778-b56a-37a06740e8ee" path="/var/lib/kubelet/pods/5438ad60-7f7b-4778-b56a-37a06740e8ee/volumes" Oct 30 01:49:19.838828 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:19.838796 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 01:49:20.129075 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.129033 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:20.129595 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.129152 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 01:49:20.429604 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.429494 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5"] Oct 30 01:49:20.430368 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.430344 5035 cpu_manager.go:414] "containerMap: removing container" podUID="aa99545f-be08-4953-8537-54935ebcceb3" containerName="payload-processing" Oct 30 01:49:20.430505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.430368 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa99545f-be08-4953-8537-54935ebcceb3" containerName="payload-processing" Oct 30 01:49:20.430505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.430389 5035 cpu_manager.go:414] "containerMap: removing container" podUID="604332d8-bef6-4ba2-85d9-65fa46b15d79" containerName="payload-pre-processing" Oct 30 01:49:20.430505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.430397 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604332d8-bef6-4ba2-85d9-65fa46b15d79" containerName="payload-pre-processing" Oct 30 01:49:20.430505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.430412 5035 cpu_manager.go:414] "containerMap: removing container" podUID="5438ad60-7f7b-4778-b56a-37a06740e8ee" containerName="maas-api" Oct 30 01:49:20.430505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.430420 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5438ad60-7f7b-4778-b56a-37a06740e8ee" containerName="maas-api" Oct 30 01:49:20.430801 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.430712 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa99545f-be08-4953-8537-54935ebcceb3" containerName="payload-processing" Oct 30 01:49:20.430801 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.430730 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="604332d8-bef6-4ba2-85d9-65fa46b15d79" containerName="payload-pre-processing" Oct 30 01:49:20.430801 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.430742 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="5438ad60-7f7b-4778-b56a-37a06740e8ee" containerName="maas-api" Oct 30 01:49:20.436273 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.436244 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.475562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.467191 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5"] Oct 30 01:49:20.485511 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.485467 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f6f4f0-openshift-default-dockercfg-jl28w\"" Oct 30 01:49:20.652628 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.652585 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.652828 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.652636 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.652828 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.652762 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcxj\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-kube-api-access-hmcxj\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.652828 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.652788 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b88633-6d70-42e1-8731-bf1024b67844-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.652947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.652867 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.652947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.652925 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.653023 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.652960 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.653023 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.652988 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b88633-6d70-42e1-8731-bf1024b67844-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.653097 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.653077 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.753985 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.753945 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b88633-6d70-42e1-8731-bf1024b67844-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.753985 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.753989 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754270 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754029 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754270 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754049 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754270 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754082 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcxj\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-kube-api-access-hmcxj\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754270 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754110 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b88633-6d70-42e1-8731-bf1024b67844-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754270 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754166 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754270 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754230 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754270 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754270 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754661 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754552 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754661 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754651 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754767 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754708 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754812 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.754896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.754832 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b88633-6d70-42e1-8731-bf1024b67844-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.756568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.756518 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.756835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.756813 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b88633-6d70-42e1-8731-bf1024b67844-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.786198 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.786155 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.789564 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.788917 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcxj\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-kube-api-access-hmcxj\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.814905 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.814872 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:20.845370 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:20.845341 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 01:49:21.129184 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:21.129095 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:21.129184 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:21.129154 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 01:49:21.162715 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:21.162683 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5"] Oct 30 01:49:21.164846 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:49:21.164814 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b88633_6d70_42e1_8731_bf1024b67844.slice/crio-56ca622c6e40322164ec6386f59829379508fa2967dbe8ac95b6523bac7d1d4b WatchSource:0}: Error finding container 56ca622c6e40322164ec6386f59829379508fa2967dbe8ac95b6523bac7d1d4b: Status 404 returned error can't find the container with id 56ca622c6e40322164ec6386f59829379508fa2967dbe8ac95b6523bac7d1d4b Oct 30 01:49:21.192135 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:21.189835 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:21.192135 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:21.189971 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:21.192135 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:21.190025 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:21.955999 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:21.955958 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" event={"ID":"e3b88633-6d70-42e1-8731-bf1024b67844","Type":"ContainerStarted","Data":"cb56ccc9a615c43183eecd359cc5d62f58caaf0e4cb7e33810cc114ed79fd261"} Oct 30 01:49:21.956219 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:21.956024 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" event={"ID":"e3b88633-6d70-42e1-8731-bf1024b67844","Type":"ContainerStarted","Data":"56ca622c6e40322164ec6386f59829379508fa2967dbe8ac95b6523bac7d1d4b"} Oct 30 01:49:22.048028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:22.047975 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" podStartSLOduration=2.047959567 podStartE2EDuration="2.047959567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:22.034770336 +0000 UTC m=+1240.537702556" watchObservedRunningTime="2026-10-30 01:49:22.047959567 +0000 UTC m=+1240.550891786" Oct 30 01:49:22.129204 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:22.129165 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:49:22.129680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:22.129230 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 30 01:49:22.815084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:22.815043 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:22.820988 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:22.820956 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:22.963448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:22.963414 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:22.964767 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:22.964739 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:23.187068 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:23.186980 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:49:23.187508 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:23.187456 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:49:23.188460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:23.188439 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 01:49:24.325495 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.325398 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:49:24.421148 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421101 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-socket\") pod \"1787c98c-df0e-41b6-bef6-e414485965d4\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " Oct 30 01:49:24.421350 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421178 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-kube-api-access-hv88w\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-kube-api-access-hv88w\") pod \"1787c98c-df0e-41b6-bef6-e414485965d4\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " Oct 30 01:49:24.421350 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421237 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-data\") pod \"1787c98c-df0e-41b6-bef6-e414485965d4\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " Oct 30 01:49:24.421350 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421264 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-envoy\") pod \"1787c98c-df0e-41b6-bef6-e414485965d4\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " Oct 30 01:49:24.421350 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421338 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-istio-token\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-istio-token\") pod \"1787c98c-df0e-41b6-bef6-e414485965d4\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " Oct 30 01:49:24.421609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421418 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-certs\") pod \"1787c98c-df0e-41b6-bef6-e414485965d4\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " Oct 30 01:49:24.421609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421426 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-socket" pod "1787c98c-df0e-41b6-bef6-e414485965d4" (UID: "1787c98c-df0e-41b6-bef6-e414485965d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:24.421609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421447 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-credential-socket\") pod \"1787c98c-df0e-41b6-bef6-e414485965d4\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " Oct 30 01:49:24.421609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421506 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1787c98c-df0e-41b6-bef6-e414485965d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1787c98c-df0e-41b6-bef6-e414485965d4-istiod-ca-cert\") pod \"1787c98c-df0e-41b6-bef6-e414485965d4\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " Oct 30 01:49:24.421609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421556 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1787c98c-df0e-41b6-bef6-e414485965d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1787c98c-df0e-41b6-bef6-e414485965d4-istio-podinfo\") pod \"1787c98c-df0e-41b6-bef6-e414485965d4\" (UID: \"1787c98c-df0e-41b6-bef6-e414485965d4\") " Oct 30 01:49:24.421609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421578 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-data" pod "1787c98c-df0e-41b6-bef6-e414485965d4" (UID: "1787c98c-df0e-41b6-bef6-e414485965d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:24.421930 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421894 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1787c98c-df0e-41b6-bef6-e414485965d4-istiod-ca-cert" pod "1787c98c-df0e-41b6-bef6-e414485965d4" (UID: "1787c98c-df0e-41b6-bef6-e414485965d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:24.422044 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.422017 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:24.422044 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.422045 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:24.422242 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.422059 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1787c98c-df0e-41b6-bef6-e414485965d4-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:24.422242 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.421740 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-credential-socket" pod "1787c98c-df0e-41b6-bef6-e414485965d4" (UID: "1787c98c-df0e-41b6-bef6-e414485965d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:24.422626 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.422594 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-certs" pod "1787c98c-df0e-41b6-bef6-e414485965d4" (UID: "1787c98c-df0e-41b6-bef6-e414485965d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:24.424026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.424003 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-envoy" pod "1787c98c-df0e-41b6-bef6-e414485965d4" (UID: "1787c98c-df0e-41b6-bef6-e414485965d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:24.424219 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.424179 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1787c98c-df0e-41b6-bef6-e414485965d4-istio-podinfo" pod "1787c98c-df0e-41b6-bef6-e414485965d4" (UID: "1787c98c-df0e-41b6-bef6-e414485965d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:49:24.424448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.424427 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-kube-api-access-hv88w" pod "1787c98c-df0e-41b6-bef6-e414485965d4" (UID: "1787c98c-df0e-41b6-bef6-e414485965d4"). InnerVolumeSpecName "kube-api-access-hv88w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:24.424636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.424613 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-istio-token" pod "1787c98c-df0e-41b6-bef6-e414485965d4" (UID: "1787c98c-df0e-41b6-bef6-e414485965d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:24.523334 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.523294 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:24.523334 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.523327 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:24.523334 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.523337 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:24.523596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.523348 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1787c98c-df0e-41b6-bef6-e414485965d4-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:24.523596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.523358 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv88w\" (UniqueName: \"kubernetes.io/projected/1787c98c-df0e-41b6-bef6-e414485965d4-kube-api-access-hv88w\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:24.523596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.523368 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1787c98c-df0e-41b6-bef6-e414485965d4-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:49:24.979594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.979557 5035 generic.go:356] "Generic (PLEG): container finished" podID="1787c98c-df0e-41b6-bef6-e414485965d4" containerID="fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b" exitCode=0 Oct 30 01:49:24.979786 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.979607 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" Oct 30 01:49:24.979786 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.979630 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" event={"ID":"1787c98c-df0e-41b6-bef6-e414485965d4","Type":"ContainerDied","Data":"fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b"} Oct 30 01:49:24.979786 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.979688 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd" event={"ID":"1787c98c-df0e-41b6-bef6-e414485965d4","Type":"ContainerDied","Data":"a32e1b3b9e449457d84e5ada57643f31d66466c66ce3ff89da972545a9d5f98d"} Oct 30 01:49:24.979786 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.979711 5035 scope.go:122] "RemoveContainer" containerID="fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b" Oct 30 01:49:24.990999 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:24.990869 5035 scope.go:122] "RemoveContainer" containerID="a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd" Oct 30 01:49:25.001699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.001675 5035 scope.go:122] "RemoveContainer" containerID="fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b" Oct 30 01:49:25.002013 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:25.001990 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b\": container with ID starting with fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b not found: ID does not exist" containerID="fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b" Oct 30 01:49:25.002110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.002027 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b"} err="failed to get container status \"fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b\": rpc error: code = NotFound desc = could not find container \"fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b\": container with ID starting with fc2bf28b98728ab1918874dfb28fb9de6bc2769df545306ae6496acc1992345b not found: ID does not exist" Oct 30 01:49:25.002110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.002053 5035 scope.go:122] "RemoveContainer" containerID="a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd" Oct 30 01:49:25.002306 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:25.002287 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd\": container with ID starting with a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd not found: ID does not exist" containerID="a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd" Oct 30 01:49:25.002358 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.002315 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd"} err="failed to get container status \"a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd\": rpc error: code = NotFound desc = could not find container \"a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd\": container with ID starting with a715cfa8b712d2532403b90d92bbaaffc03f870190f39ed7c2435a9fce6399fd not found: ID does not exist" Oct 30 01:49:25.057327 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.057287 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd"] Oct 30 01:49:25.070512 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.070475 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6c3505dc-openshift-default-66b5c6c749-bm4hd"] Oct 30 01:49:25.370747 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.370662 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z"] Oct 30 01:49:25.371195 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.371180 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" Oct 30 01:49:25.371241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.371196 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" Oct 30 01:49:25.371241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.371218 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" Oct 30 01:49:25.371241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.371226 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" Oct 30 01:49:25.371326 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.371317 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" Oct 30 01:49:25.376246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.376218 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.395497 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.395460 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"openshift-service-ca.crt\"" Oct 30 01:49:25.395720 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.395590 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"kube-root-ca.crt\"" Oct 30 01:49:25.395795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.395719 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"default-dockercfg-wr9nk\"" Oct 30 01:49:25.395795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.395721 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"simulator-w2-719029-kserve-self-signed-certs\"" Oct 30 01:49:25.399301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.399272 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z"] Oct 30 01:49:25.433028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.432981 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-dshm\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.433028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.433031 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388a3149-77ea-469e-8038-681c4f10cc37-tls-certs\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.433310 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.433184 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-model-cache\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.433310 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.433219 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bsbh\" (UniqueName: \"kubernetes.io/projected/388a3149-77ea-469e-8038-681c4f10cc37-kube-api-access-8bsbh\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.433418 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.433305 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-home\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.433418 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.433372 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-tmp-dir\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.534952 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.534909 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-model-cache\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.534952 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.534949 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bsbh\" (UniqueName: \"kubernetes.io/projected/388a3149-77ea-469e-8038-681c4f10cc37-kube-api-access-8bsbh\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.535179 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.534972 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-home\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.535179 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.534998 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-tmp-dir\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.535275 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.535145 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-dshm\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.535275 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.535221 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388a3149-77ea-469e-8038-681c4f10cc37-tls-certs\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.535516 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.535454 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-model-cache\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.535733 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.535700 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-home\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.535918 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.535883 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-tmp-dir\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.537792 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.537760 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-dshm\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.537945 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.537926 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388a3149-77ea-469e-8038-681c4f10cc37-tls-certs\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.559573 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.559517 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bsbh\" (UniqueName: \"kubernetes.io/projected/388a3149-77ea-469e-8038-681c4f10cc37-kube-api-access-8bsbh\") pod \"simulator-w2-719029-kserve-6b5544794f-vnf6z\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:25.645913 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.645820 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" path="/var/lib/kubelet/pods/1787c98c-df0e-41b6-bef6-e414485965d4/volumes" Oct 30 01:49:25.689770 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:25.689732 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:26.011251 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.008575 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z"] Oct 30 01:49:26.015734 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:49:26.015698 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod388a3149_77ea_469e_8038_681c4f10cc37.slice/crio-dd017b63b127254cecf5719574e4ebaea0f8eb56e8719c54b752d0ea0299595c WatchSource:0}: Error finding container dd017b63b127254cecf5719574e4ebaea0f8eb56e8719c54b752d0ea0299595c: Status 404 returned error can't find the container with id dd017b63b127254cecf5719574e4ebaea0f8eb56e8719c54b752d0ea0299595c Oct 30 01:49:26.674381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.674333 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz"] Oct 30 01:49:26.675186 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.675159 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1787c98c-df0e-41b6-bef6-e414485965d4" containerName="istio-proxy" Oct 30 01:49:26.678686 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.678662 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:26.696919 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.696884 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz"] Oct 30 01:49:26.706998 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.706967 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f6f4f0\"" Oct 30 01:49:26.707172 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.707003 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f6f4f0-dockercfg-x5v2j\"" Oct 30 01:49:26.753186 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.753107 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d34e97-6a00-4733-8296-f16c73833283-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz\" (UID: \"e2d34e97-6a00-4733-8296-f16c73833283\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:26.753451 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.753217 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmfz\" (UniqueName: \"kubernetes.io/projected/e2d34e97-6a00-4733-8296-f16c73833283-kube-api-access-fkmfz\") pod \"payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz\" (UID: \"e2d34e97-6a00-4733-8296-f16c73833283\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:26.788627 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.788577 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z"] Oct 30 01:49:26.792804 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.792776 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:26.841093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.841058 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z"] Oct 30 01:49:26.854201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.854158 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d34e97-6a00-4733-8296-f16c73833283-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz\" (UID: \"e2d34e97-6a00-4733-8296-f16c73833283\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:26.854380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.854234 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmfz\" (UniqueName: \"kubernetes.io/projected/e2d34e97-6a00-4733-8296-f16c73833283-kube-api-access-fkmfz\") pod \"payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz\" (UID: \"e2d34e97-6a00-4733-8296-f16c73833283\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:26.854380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.854296 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4g9p\" (UniqueName: \"kubernetes.io/projected/957eece0-9249-4035-ad6b-a8e2a967e3a8-kube-api-access-l4g9p\") pod \"payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z\" (UID: \"957eece0-9249-4035-ad6b-a8e2a967e3a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:26.854380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.854361 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957eece0-9249-4035-ad6b-a8e2a967e3a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z\" (UID: \"957eece0-9249-4035-ad6b-a8e2a967e3a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:26.855084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.855063 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d34e97-6a00-4733-8296-f16c73833283-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz\" (UID: \"e2d34e97-6a00-4733-8296-f16c73833283\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:26.881459 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.881426 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmfz\" (UniqueName: \"kubernetes.io/projected/e2d34e97-6a00-4733-8296-f16c73833283-kube-api-access-fkmfz\") pod \"payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz\" (UID: \"e2d34e97-6a00-4733-8296-f16c73833283\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:26.955145 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.955107 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4g9p\" (UniqueName: \"kubernetes.io/projected/957eece0-9249-4035-ad6b-a8e2a967e3a8-kube-api-access-l4g9p\") pod \"payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z\" (UID: \"957eece0-9249-4035-ad6b-a8e2a967e3a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:26.955308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.955183 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957eece0-9249-4035-ad6b-a8e2a967e3a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z\" (UID: \"957eece0-9249-4035-ad6b-a8e2a967e3a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:26.955990 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.955969 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957eece0-9249-4035-ad6b-a8e2a967e3a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z\" (UID: \"957eece0-9249-4035-ad6b-a8e2a967e3a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:26.981134 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.981090 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4g9p\" (UniqueName: \"kubernetes.io/projected/957eece0-9249-4035-ad6b-a8e2a967e3a8-kube-api-access-l4g9p\") pod \"payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z\" (UID: \"957eece0-9249-4035-ad6b-a8e2a967e3a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:26.993235 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:26.993198 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:27.009110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:27.009069 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" event={"ID":"388a3149-77ea-469e-8038-681c4f10cc37","Type":"ContainerStarted","Data":"d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58"} Oct 30 01:49:27.009238 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:27.009132 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" event={"ID":"388a3149-77ea-469e-8038-681c4f10cc37","Type":"ContainerStarted","Data":"dd017b63b127254cecf5719574e4ebaea0f8eb56e8719c54b752d0ea0299595c"} Oct 30 01:49:27.056804 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:27.055847 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podStartSLOduration=2.055824541 podStartE2EDuration="2.055824541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:27.053436197 +0000 UTC m=+1245.556368418" watchObservedRunningTime="2026-10-30 01:49:27.055824541 +0000 UTC m=+1245.558756762" Oct 30 01:49:27.107597 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:27.107511 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:27.296822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:27.296794 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz"] Oct 30 01:49:27.298816 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:49:27.298783 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d34e97_6a00_4733_8296_f16c73833283.slice/crio-d07e2e029ea6c3969136d9c4d36c7f0de6d7ec926864770601b3b52b9d680b85 WatchSource:0}: Error finding container d07e2e029ea6c3969136d9c4d36c7f0de6d7ec926864770601b3b52b9d680b85: Status 404 returned error can't find the container with id d07e2e029ea6c3969136d9c4d36c7f0de6d7ec926864770601b3b52b9d680b85 Oct 30 01:49:27.408203 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:27.408176 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z"] Oct 30 01:49:27.410421 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:49:27.410393 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957eece0_9249_4035_ad6b_a8e2a967e3a8.slice/crio-1a303971e6f538f3b0a963a751ba2828204c7c6fd75e53285fa0bef579b4f8b2 WatchSource:0}: Error finding container 1a303971e6f538f3b0a963a751ba2828204c7c6fd75e53285fa0bef579b4f8b2: Status 404 returned error can't find the container with id 1a303971e6f538f3b0a963a751ba2828204c7c6fd75e53285fa0bef579b4f8b2 Oct 30 01:49:27.690502 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:27.690409 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:28.017717 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:28.017672 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" event={"ID":"957eece0-9249-4035-ad6b-a8e2a967e3a8","Type":"ContainerStarted","Data":"1a99f001f01854d01f8f91485020ce691c183871745b8a4e54bdd6dd165b0a79"} Oct 30 01:49:28.017891 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:28.017742 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" event={"ID":"957eece0-9249-4035-ad6b-a8e2a967e3a8","Type":"ContainerStarted","Data":"1a303971e6f538f3b0a963a751ba2828204c7c6fd75e53285fa0bef579b4f8b2"} Oct 30 01:49:28.017891 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:28.017874 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:28.019429 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:28.019397 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" event={"ID":"e2d34e97-6a00-4733-8296-f16c73833283","Type":"ContainerStarted","Data":"a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317"} Oct 30 01:49:28.019630 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:28.019559 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:28.019630 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:28.019600 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" event={"ID":"e2d34e97-6a00-4733-8296-f16c73833283","Type":"ContainerStarted","Data":"d07e2e029ea6c3969136d9c4d36c7f0de6d7ec926864770601b3b52b9d680b85"} Oct 30 01:49:28.067690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:28.067631 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" podStartSLOduration=2.067616097 podStartE2EDuration="2.067616097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:28.06514168 +0000 UTC m=+1246.568073928" watchObservedRunningTime="2026-10-30 01:49:28.067616097 +0000 UTC m=+1246.570548355" Oct 30 01:49:28.107998 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:28.107944 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" podStartSLOduration=2.107927956 podStartE2EDuration="2.107927956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:28.10751528 +0000 UTC m=+1246.610447499" watchObservedRunningTime="2026-10-30 01:49:28.107927956 +0000 UTC m=+1246.610860176" Oct 30 01:49:34.902768 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:34.902729 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2"] Oct 30 01:49:34.907908 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:34.907878 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:34.948873 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:34.945783 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-f6f4f0-serving-cert\"" Oct 30 01:49:34.948873 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:34.945931 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:49:34.948873 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:34.945951 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:49:34.948873 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:34.946039 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 01:49:34.948873 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:34.946478 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xzt6k\"" Oct 30 01:49:34.948873 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:34.948856 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2"] Oct 30 01:49:35.047666 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.047620 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-metrics-certs\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.047851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.047746 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.047893 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.047834 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cae86e-0335-4f80-8521-fec459fe858f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.047938 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.047907 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclcx\" (UniqueName: \"kubernetes.io/projected/e4cae86e-0335-4f80-8521-fec459fe858f-kube-api-access-jclcx\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.149676 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.149635 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-metrics-certs\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.149874 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.149810 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.149914 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.149870 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cae86e-0335-4f80-8521-fec459fe858f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.149951 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.149916 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclcx\" (UniqueName: \"kubernetes.io/projected/e4cae86e-0335-4f80-8521-fec459fe858f-kube-api-access-jclcx\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.149951 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:35.149925 5035 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-serving-cert: secret "maas-api-e2e-worker-w1-f6f4f0-serving-cert" not found Oct 30 01:49:35.150015 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:35.149993 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls podName:e4cae86e-0335-4f80-8521-fec459fe858f nodeName:}" failed. No retries permitted until 2026-10-30 01:49:35.649976204 +0000 UTC m=+1254.152908433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls") pod "maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" (UID: "e4cae86e-0335-4f80-8521-fec459fe858f") : secret "maas-api-e2e-worker-w1-f6f4f0-serving-cert" not found Oct 30 01:49:35.150612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.150587 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cae86e-0335-4f80-8521-fec459fe858f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.152335 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.152308 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-metrics-certs\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.177681 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.177591 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclcx\" (UniqueName: \"kubernetes.io/projected/e4cae86e-0335-4f80-8521-fec459fe858f-kube-api-access-jclcx\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.656026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.655991 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.658561 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.658496 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:35.689941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.689880 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:35.702916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.702878 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:35.822490 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:35.822454 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:36.084488 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:36.084454 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:49:36.156410 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:36.156364 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2"] Oct 30 01:49:37.080420 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:37.080378 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" event={"ID":"e4cae86e-0335-4f80-8521-fec459fe858f","Type":"ContainerStarted","Data":"184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90"} Oct 30 01:49:37.080673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:37.080446 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" event={"ID":"e4cae86e-0335-4f80-8521-fec459fe858f","Type":"ContainerStarted","Data":"3e7f98f13c06ac3d5d47597d61d43669e7adeaa39954c8c51da511e21cc0eba7"} Oct 30 01:49:37.080673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:37.080608 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:37.127954 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:37.127888 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" podStartSLOduration=2.798857232 podStartE2EDuration="3.127870887s" totalImagesPullingTime="329.013655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:37.125388749 +0000 UTC m=+1255.628320969" watchObservedRunningTime="2026-10-30 01:49:37.127870887 +0000 UTC m=+1255.630803106" Oct 30 01:49:39.028577 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:39.028519 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 01:49:39.028995 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:39.028715 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 01:49:43.099615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:43.099586 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 01:49:47.630051 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:49:47.630008 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:49:49.158175 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:49.158086 5035 generic.go:356] "Generic (PLEG): container finished" podID="e3b88633-6d70-42e1-8731-bf1024b67844" containerID="cb56ccc9a615c43183eecd359cc5d62f58caaf0e4cb7e33810cc114ed79fd261" exitCode=0 Oct 30 01:49:49.158175 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:49.158139 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" event={"ID":"e3b88633-6d70-42e1-8731-bf1024b67844","Type":"ContainerDied","Data":"cb56ccc9a615c43183eecd359cc5d62f58caaf0e4cb7e33810cc114ed79fd261"} Oct 30 01:49:49.158728 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:49.158583 5035 scope.go:122] "RemoveContainer" containerID="cb56ccc9a615c43183eecd359cc5d62f58caaf0e4cb7e33810cc114ed79fd261" Oct 30 01:49:49.160014 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:49.159980 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:49.160077 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:49.160060 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:49.160117 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:49.160091 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:50.176289 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:50.176230 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" event={"ID":"e3b88633-6d70-42e1-8731-bf1024b67844","Type":"ContainerStarted","Data":"43a868e538322d0c1483f42e9e9567a32b97439a56a28a6d311781bf9bc86309"} Oct 30 01:49:50.816163 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:50.816122 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:50.817291 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:50.817262 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:49:50.817422 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:50.817319 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 01:49:51.816106 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:51.816067 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:49:51.816650 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:51.816128 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 01:49:52.815357 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:52.815317 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:49:52.815585 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:52.815377 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 01:49:52.964103 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:52.964055 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:53.815704 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:53.815669 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:49:53.815886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:53.815752 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 01:49:54.815771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:54.815726 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:49:54.816290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:54.815819 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 01:49:55.820231 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:55.820198 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:55.821396 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:55.821372 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:49:56.519887 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.519849 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946"] Oct 30 01:49:56.523248 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.523221 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.534319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.533657 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxhvb\" (UniqueName: \"kubernetes.io/projected/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-kube-api-access-zxhvb\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.534319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.533716 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-home\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.534319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.533815 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-dshm\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.534319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.533856 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-model-cache\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.534319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.533900 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tmp-dir\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.534319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.533942 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tls-certs\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.555845 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.555810 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"premium-w2-719029-kserve-self-signed-certs\"" Oct 30 01:49:56.558738 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.558713 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946"] Oct 30 01:49:56.634797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.634745 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tmp-dir\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.635001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.634819 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tls-certs\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.635001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.634875 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhvb\" (UniqueName: \"kubernetes.io/projected/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-kube-api-access-zxhvb\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.635001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.634913 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-home\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.635131 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.635025 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-dshm\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.635131 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.635075 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-model-cache\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.635243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.635220 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tmp-dir\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.635385 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.635357 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-home\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.635448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.635387 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-model-cache\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.637349 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.637315 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-dshm\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.637487 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.637468 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tls-certs\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.660613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.660576 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxhvb\" (UniqueName: \"kubernetes.io/projected/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-kube-api-access-zxhvb\") pod \"premium-w2-719029-kserve-6c6b979dc6-6h946\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:56.839473 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:56.839372 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:49:57.142489 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:57.142463 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946"] Oct 30 01:49:57.144600 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:49:57.144568 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d83b6a_03b9_4fd9_8fa9_74cf8d8e7ace.slice/crio-b648122c6d8033fdd3189f2615146e5bfd88794d23128faafca043b0f9491dac WatchSource:0}: Error finding container b648122c6d8033fdd3189f2615146e5bfd88794d23128faafca043b0f9491dac: Status 404 returned error can't find the container with id b648122c6d8033fdd3189f2615146e5bfd88794d23128faafca043b0f9491dac Oct 30 01:49:57.242994 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:57.242949 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" event={"ID":"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace","Type":"ContainerStarted","Data":"c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af"} Oct 30 01:49:57.243176 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:57.243012 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" event={"ID":"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace","Type":"ContainerStarted","Data":"b648122c6d8033fdd3189f2615146e5bfd88794d23128faafca043b0f9491dac"} Oct 30 01:49:57.287384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:57.287321 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podStartSLOduration=1.287304542 podStartE2EDuration="1.287304542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:57.287052925 +0000 UTC m=+1275.789985136" watchObservedRunningTime="2026-10-30 01:49:57.287304542 +0000 UTC m=+1275.790236762" Oct 30 01:49:57.839775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:49:57.839738 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:50:06.840302 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:06.840252 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:50:06.853202 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:06.853168 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:50:07.311185 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:07.311154 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:50:12.017622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.017585 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw"] Oct 30 01:50:12.022783 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.022758 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.042713 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.042680 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw"] Oct 30 01:50:12.053013 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.052965 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"openshift-service-ca.crt\"" Oct 30 01:50:12.053210 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.053033 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"kube-root-ca.crt\"" Oct 30 01:50:12.053210 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.053096 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"default-dockercfg-cj96z\"" Oct 30 01:50:12.055280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.055261 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"premium-w4-b93d51-kserve-self-signed-certs\"" Oct 30 01:50:12.096123 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.096073 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-dshm\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.096123 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.096122 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-model-cache\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.096356 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.096141 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-home\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.096356 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.096270 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0590d679-8944-4559-a287-07a90e6d0f42-tls-certs\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.096448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.096369 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22klc\" (UniqueName: \"kubernetes.io/projected/0590d679-8944-4559-a287-07a90e6d0f42-kube-api-access-22klc\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.096502 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.096483 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-tmp-dir\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.197610 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.197565 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-tmp-dir\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.197867 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.197716 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-dshm\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.197867 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.197752 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-model-cache\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.197867 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.197781 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-home\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.197867 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.197840 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0590d679-8944-4559-a287-07a90e6d0f42-tls-certs\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.198108 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.197947 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22klc\" (UniqueName: \"kubernetes.io/projected/0590d679-8944-4559-a287-07a90e6d0f42-kube-api-access-22klc\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.199491 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.199462 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-model-cache\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.199812 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.199790 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-tmp-dir\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.202560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.202486 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-home\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.203019 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.202995 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-dshm\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.226407 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.226371 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0590d679-8944-4559-a287-07a90e6d0f42-tls-certs\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.239396 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.239363 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22klc\" (UniqueName: \"kubernetes.io/projected/0590d679-8944-4559-a287-07a90e6d0f42-kube-api-access-22klc\") pod \"premium-w4-b93d51-kserve-65788c994b-jbzmw\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.336452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.336312 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:12.661103 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:12.660411 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw"] Oct 30 01:50:12.662985 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:50:12.662951 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0590d679_8944_4559_a287_07a90e6d0f42.slice/crio-5fffa7e944c121b2cf7294d37a4ed3f58ccf21e0ca39ee0aa92fa084b619d3e7 WatchSource:0}: Error finding container 5fffa7e944c121b2cf7294d37a4ed3f58ccf21e0ca39ee0aa92fa084b619d3e7: Status 404 returned error can't find the container with id 5fffa7e944c121b2cf7294d37a4ed3f58ccf21e0ca39ee0aa92fa084b619d3e7 Oct 30 01:50:13.336955 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:13.336915 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" event={"ID":"0590d679-8944-4559-a287-07a90e6d0f42","Type":"ContainerStarted","Data":"4a60d10c03adeb9b8b4c78761e2dedb503f90e9b982863df66f1799b6db77b4c"} Oct 30 01:50:13.336955 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:13.336966 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" event={"ID":"0590d679-8944-4559-a287-07a90e6d0f42","Type":"ContainerStarted","Data":"5fffa7e944c121b2cf7294d37a4ed3f58ccf21e0ca39ee0aa92fa084b619d3e7"} Oct 30 01:50:13.384353 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:13.384287 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podStartSLOduration=2.384269501 podStartE2EDuration="2.384269501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:13.382744426 +0000 UTC m=+1291.885676648" watchObservedRunningTime="2026-10-30 01:50:13.384269501 +0000 UTC m=+1291.887201727" Oct 30 01:50:14.336421 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:14.336380 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:17.628653 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:50:17.628608 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:50:22.337109 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:22.337064 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:22.350234 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:22.350202 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:22.405574 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:22.405507 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:50:29.486997 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.486912 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh"] Oct 30 01:50:29.491475 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.491452 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.498435 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.498401 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4r8\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-kube-api-access-rz4r8\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.498607 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.498444 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.498607 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.498570 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.498686 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.498624 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.498720 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.498688 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.498770 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.498758 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.498806 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.498793 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.498839 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.498818 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.498911 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.498887 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.521201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.521170 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b93d51-openshift-default-dockercfg-qh2lt\"" Oct 30 01:50:29.524985 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.524945 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh"] Oct 30 01:50:29.599421 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599363 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.599681 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599442 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.599681 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599491 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.599681 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599599 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.599681 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599638 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.599681 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599673 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.599973 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599716 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.599973 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599758 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4r8\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-kube-api-access-rz4r8\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.599973 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599792 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.600137 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.599992 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.600137 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.600045 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.600395 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.600367 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.600497 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.600394 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.600631 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.600612 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.602133 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.602106 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.602253 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.602225 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.624857 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.624816 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.625012 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.624935 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4r8\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-kube-api-access-rz4r8\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:29.805587 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:29.805466 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:30.131271 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:30.131240 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh"] Oct 30 01:50:30.132763 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:50:30.132734 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0070f21f_823c_45ab_8264_39ba6c3f3c9e.slice/crio-f9f1569934dfb7e7d62450570cc1732bdc6bb5365dbec25f29439119da769f9f WatchSource:0}: Error finding container f9f1569934dfb7e7d62450570cc1732bdc6bb5365dbec25f29439119da769f9f: Status 404 returned error can't find the container with id f9f1569934dfb7e7d62450570cc1732bdc6bb5365dbec25f29439119da769f9f Oct 30 01:50:30.135162 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:30.135130 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:50:30.135243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:30.135205 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:50:30.135243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:30.135238 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:50:30.445060 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:30.444955 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" event={"ID":"0070f21f-823c-45ab-8264-39ba6c3f3c9e","Type":"ContainerStarted","Data":"a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218"} Oct 30 01:50:30.445060 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:30.445021 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" event={"ID":"0070f21f-823c-45ab-8264-39ba6c3f3c9e","Type":"ContainerStarted","Data":"f9f1569934dfb7e7d62450570cc1732bdc6bb5365dbec25f29439119da769f9f"} Oct 30 01:50:30.493987 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:30.493929 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" podStartSLOduration=1.493911254 podStartE2EDuration="1.493911254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:30.492827991 +0000 UTC m=+1308.995760212" watchObservedRunningTime="2026-10-30 01:50:30.493911254 +0000 UTC m=+1308.996843474" Oct 30 01:50:30.806361 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:30.806318 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:31.808105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:31.808068 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:50:31.808561 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:31.808145 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:50:32.806338 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:32.806295 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:50:32.806508 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:32.806365 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:50:33.806673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:33.806630 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:50:33.807199 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:33.806699 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:50:34.806042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:34.806000 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:50:34.806243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:34.806068 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:50:35.806170 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:35.806130 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:50:35.806628 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:35.806228 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:50:36.968142 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:36.968099 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:36.968754 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:36.968723 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:36.970423 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:36.970394 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 01:50:47.349425 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.349376 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8"] Oct 30 01:50:47.354785 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.354760 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.375130 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.375093 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8"] Oct 30 01:50:47.384559 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.384491 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:50:47.384760 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.384593 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:50:47.444594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.444559 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da600374-f947-4104-906d-aa030d698e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e81ef2c-nc8h8\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.444766 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.444654 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da600374-f947-4104-906d-aa030d698e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e81ef2c-nc8h8\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.444811 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.444761 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvct\" (UniqueName: \"kubernetes.io/projected/da600374-f947-4104-906d-aa030d698e51-kube-api-access-9zvct\") pod \"maas-api-revoke-sub-4e81ef2c-nc8h8\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.545428 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.545386 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da600374-f947-4104-906d-aa030d698e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e81ef2c-nc8h8\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.545676 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.545485 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da600374-f947-4104-906d-aa030d698e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e81ef2c-nc8h8\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.545676 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.545517 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvct\" (UniqueName: \"kubernetes.io/projected/da600374-f947-4104-906d-aa030d698e51-kube-api-access-9zvct\") pod \"maas-api-revoke-sub-4e81ef2c-nc8h8\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.546275 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.546252 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da600374-f947-4104-906d-aa030d698e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e81ef2c-nc8h8\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.547901 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.547872 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da600374-f947-4104-906d-aa030d698e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e81ef2c-nc8h8\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.570099 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.570065 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvct\" (UniqueName: \"kubernetes.io/projected/da600374-f947-4104-906d-aa030d698e51-kube-api-access-9zvct\") pod \"maas-api-revoke-sub-4e81ef2c-nc8h8\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.629313 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:50:47.629220 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:50:47.669635 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.669594 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:47.973309 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:47.973265 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8"] Oct 30 01:50:47.975105 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:50:47.975071 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda600374_f947_4104_906d_aa030d698e51.slice/crio-7055006791ef232b5c069616cc6e37f6fc6dea4ee5c092e4476f4717203f0d59 WatchSource:0}: Error finding container 7055006791ef232b5c069616cc6e37f6fc6dea4ee5c092e4476f4717203f0d59: Status 404 returned error can't find the container with id 7055006791ef232b5c069616cc6e37f6fc6dea4ee5c092e4476f4717203f0d59 Oct 30 01:50:48.602118 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:48.602004 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" event={"ID":"da600374-f947-4104-906d-aa030d698e51","Type":"ContainerStarted","Data":"78657004f70f4de664934c158b96a1a1851ba5baf20231472613f5986090cc43"} Oct 30 01:50:48.602118 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:48.602067 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" event={"ID":"da600374-f947-4104-906d-aa030d698e51","Type":"ContainerStarted","Data":"7055006791ef232b5c069616cc6e37f6fc6dea4ee5c092e4476f4717203f0d59"} Oct 30 01:50:48.646481 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:48.646421 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" podStartSLOduration=1.646404425 podStartE2EDuration="1.646404425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:48.644394978 +0000 UTC m=+1327.147327524" watchObservedRunningTime="2026-10-30 01:50:48.646404425 +0000 UTC m=+1327.149336645" Oct 30 01:50:49.610340 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:49.610298 5035 generic.go:356] "Generic (PLEG): container finished" podID="da600374-f947-4104-906d-aa030d698e51" containerID="78657004f70f4de664934c158b96a1a1851ba5baf20231472613f5986090cc43" exitCode=0 Oct 30 01:50:49.610821 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:49.610353 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" event={"ID":"da600374-f947-4104-906d-aa030d698e51","Type":"ContainerDied","Data":"78657004f70f4de664934c158b96a1a1851ba5baf20231472613f5986090cc43"} Oct 30 01:50:50.765322 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.765293 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:50.883993 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.883898 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da600374-f947-4104-906d-aa030d698e51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da600374-f947-4104-906d-aa030d698e51-maas-api-xks-ca\") pod \"da600374-f947-4104-906d-aa030d698e51\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " Oct 30 01:50:50.883993 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.883989 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da600374-f947-4104-906d-aa030d698e51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da600374-f947-4104-906d-aa030d698e51-maas-api-openshift-ca\") pod \"da600374-f947-4104-906d-aa030d698e51\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " Oct 30 01:50:50.884188 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.884020 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da600374-f947-4104-906d-aa030d698e51-kube-api-access-9zvct\" (UniqueName: \"kubernetes.io/projected/da600374-f947-4104-906d-aa030d698e51-kube-api-access-9zvct\") pod \"da600374-f947-4104-906d-aa030d698e51\" (UID: \"da600374-f947-4104-906d-aa030d698e51\") " Oct 30 01:50:50.884443 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.884415 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da600374-f947-4104-906d-aa030d698e51-maas-api-openshift-ca" pod "da600374-f947-4104-906d-aa030d698e51" (UID: "da600374-f947-4104-906d-aa030d698e51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:50:50.884570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.884518 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da600374-f947-4104-906d-aa030d698e51-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:50:50.886148 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.886122 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da600374-f947-4104-906d-aa030d698e51-maas-api-xks-ca" pod "da600374-f947-4104-906d-aa030d698e51" (UID: "da600374-f947-4104-906d-aa030d698e51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:50:50.886207 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.886183 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da600374-f947-4104-906d-aa030d698e51-kube-api-access-9zvct" pod "da600374-f947-4104-906d-aa030d698e51" (UID: "da600374-f947-4104-906d-aa030d698e51"). InnerVolumeSpecName "kube-api-access-9zvct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:50:50.986177 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.986133 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da600374-f947-4104-906d-aa030d698e51-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:50:50.986177 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:50.986174 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvct\" (UniqueName: \"kubernetes.io/projected/da600374-f947-4104-906d-aa030d698e51-kube-api-access-9zvct\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.623710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:51.623676 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" Oct 30 01:50:51.623927 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:51.623706 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8" event={"ID":"da600374-f947-4104-906d-aa030d698e51","Type":"ContainerDied","Data":"7055006791ef232b5c069616cc6e37f6fc6dea4ee5c092e4476f4717203f0d59"} Oct 30 01:50:51.623927 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:51.623761 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7055006791ef232b5c069616cc6e37f6fc6dea4ee5c092e4476f4717203f0d59" Oct 30 01:50:59.629908 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.629860 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n"] Oct 30 01:50:59.630696 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.630511 5035 cpu_manager.go:414] "containerMap: removing container" podUID="da600374-f947-4104-906d-aa030d698e51" containerName="revoke-keys" Oct 30 01:50:59.630696 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.630545 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da600374-f947-4104-906d-aa030d698e51" containerName="revoke-keys" Oct 30 01:50:59.630844 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.630701 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="da600374-f947-4104-906d-aa030d698e51" containerName="revoke-keys" Oct 30 01:50:59.640028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.639991 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.654846 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.654814 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n"] Oct 30 01:50:59.661584 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.661518 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:50:59.661584 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.661568 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:50:59.679688 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.679643 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l4lw\" (UniqueName: \"kubernetes.io/projected/9c463943-87e6-4dfc-b83b-7ca69301fd35-kube-api-access-5l4lw\") pod \"maas-api-revoke-sub-fb3fdffa-kb28n\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.679898 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.679823 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb3fdffa-kb28n\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.680017 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.679995 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb3fdffa-kb28n\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.780638 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.780590 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb3fdffa-kb28n\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.780638 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.780646 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l4lw\" (UniqueName: \"kubernetes.io/projected/9c463943-87e6-4dfc-b83b-7ca69301fd35-kube-api-access-5l4lw\") pod \"maas-api-revoke-sub-fb3fdffa-kb28n\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.780926 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.780730 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb3fdffa-kb28n\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.781327 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.781300 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb3fdffa-kb28n\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.783079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.783056 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb3fdffa-kb28n\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.804825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.804787 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l4lw\" (UniqueName: \"kubernetes.io/projected/9c463943-87e6-4dfc-b83b-7ca69301fd35-kube-api-access-5l4lw\") pod \"maas-api-revoke-sub-fb3fdffa-kb28n\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:50:59.960014 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:50:59.959979 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:51:00.264581 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:00.264545 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n"] Oct 30 01:51:00.266199 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:51:00.266164 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c463943_87e6_4dfc_b83b_7ca69301fd35.slice/crio-1cc784af14e73fb1cb7b1b7953485c646602dcb043eec83742ec894d925a68d7 WatchSource:0}: Error finding container 1cc784af14e73fb1cb7b1b7953485c646602dcb043eec83742ec894d925a68d7: Status 404 returned error can't find the container with id 1cc784af14e73fb1cb7b1b7953485c646602dcb043eec83742ec894d925a68d7 Oct 30 01:51:00.694194 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:00.694141 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" event={"ID":"9c463943-87e6-4dfc-b83b-7ca69301fd35","Type":"ContainerStarted","Data":"44e1bc1c48f502c932bb85f1e9be7ccb2c1f334b32ad21f21074d4677af8ae4d"} Oct 30 01:51:00.694194 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:00.694213 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" event={"ID":"9c463943-87e6-4dfc-b83b-7ca69301fd35","Type":"ContainerStarted","Data":"1cc784af14e73fb1cb7b1b7953485c646602dcb043eec83742ec894d925a68d7"} Oct 30 01:51:00.740479 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:00.740420 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" podStartSLOduration=1.740405789 podStartE2EDuration="1.740405789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:00.738672633 +0000 UTC m=+1339.241604882" watchObservedRunningTime="2026-10-30 01:51:00.740405789 +0000 UTC m=+1339.243338009" Oct 30 01:51:01.701954 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.701919 5035 generic.go:356] "Generic (PLEG): container finished" podID="9c463943-87e6-4dfc-b83b-7ca69301fd35" containerID="44e1bc1c48f502c932bb85f1e9be7ccb2c1f334b32ad21f21074d4677af8ae4d" exitCode=0 Oct 30 01:51:01.702365 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.701998 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" event={"ID":"9c463943-87e6-4dfc-b83b-7ca69301fd35","Type":"ContainerDied","Data":"44e1bc1c48f502c932bb85f1e9be7ccb2c1f334b32ad21f21074d4677af8ae4d"} Oct 30 01:51:01.784150 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.784107 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv"] Oct 30 01:51:01.788154 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.788127 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.802382 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.802346 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.802571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.802487 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.802660 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.802638 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.802726 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.802688 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.802805 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.802791 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.802862 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.802826 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c87830-ad96-4191-b4c3-22a6c537f46d-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.802919 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.802877 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.802919 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.802905 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc2rc\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-kube-api-access-bc2rc\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.803017 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.802968 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.810876 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.807282 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c40a89-openshift-default-dockercfg-vvdq4\"" Oct 30 01:51:01.811489 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.811456 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv"] Oct 30 01:51:01.904000 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.903959 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904000 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904003 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c87830-ad96-4191-b4c3-22a6c537f46d-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904235 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904048 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904235 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904073 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc2rc\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-kube-api-access-bc2rc\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904235 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904112 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904235 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904146 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904235 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904234 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904517 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904351 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904517 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904389 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904517 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904474 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904574 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904831 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904808 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904858 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.904988 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.904959 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c87830-ad96-4191-b4c3-22a6c537f46d-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.906704 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.906673 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.906818 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.906769 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.929006 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.928965 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc2rc\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-kube-api-access-bc2rc\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:01.930484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:01.930463 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:02.102965 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.102863 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:02.422346 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.422313 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv"] Oct 30 01:51:02.423305 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:51:02.423276 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c87830_ad96_4191_b4c3_22a6c537f46d.slice/crio-8e4e4574a464cfdea91b0e78c2329447c033ca597f02682decf540e64c82a6fe WatchSource:0}: Error finding container 8e4e4574a464cfdea91b0e78c2329447c033ca597f02682decf540e64c82a6fe: Status 404 returned error can't find the container with id 8e4e4574a464cfdea91b0e78c2329447c033ca597f02682decf540e64c82a6fe Oct 30 01:51:02.425829 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.425793 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:51:02.425916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.425866 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:51:02.425916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.425896 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:51:02.711842 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.711791 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" event={"ID":"15c87830-ad96-4191-b4c3-22a6c537f46d","Type":"ContainerStarted","Data":"e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba"} Oct 30 01:51:02.712343 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.711864 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" event={"ID":"15c87830-ad96-4191-b4c3-22a6c537f46d","Type":"ContainerStarted","Data":"8e4e4574a464cfdea91b0e78c2329447c033ca597f02682decf540e64c82a6fe"} Oct 30 01:51:02.762025 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.761960 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" podStartSLOduration=1.761939015 podStartE2EDuration="1.761939015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:02.760130635 +0000 UTC m=+1341.263062856" watchObservedRunningTime="2026-10-30 01:51:02.761939015 +0000 UTC m=+1341.264871235" Oct 30 01:51:02.841749 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.841719 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:51:02.917693 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.917650 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-openshift-ca\") pod \"9c463943-87e6-4dfc-b83b-7ca69301fd35\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " Oct 30 01:51:02.917860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.917713 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-xks-ca\") pod \"9c463943-87e6-4dfc-b83b-7ca69301fd35\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " Oct 30 01:51:02.917962 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.917917 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c463943-87e6-4dfc-b83b-7ca69301fd35-kube-api-access-5l4lw\" (UniqueName: \"kubernetes.io/projected/9c463943-87e6-4dfc-b83b-7ca69301fd35-kube-api-access-5l4lw\") pod \"9c463943-87e6-4dfc-b83b-7ca69301fd35\" (UID: \"9c463943-87e6-4dfc-b83b-7ca69301fd35\") " Oct 30 01:51:02.918100 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.918068 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-openshift-ca" pod "9c463943-87e6-4dfc-b83b-7ca69301fd35" (UID: "9c463943-87e6-4dfc-b83b-7ca69301fd35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:51:02.918414 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.918390 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:51:02.920803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.920768 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-xks-ca" pod "9c463943-87e6-4dfc-b83b-7ca69301fd35" (UID: "9c463943-87e6-4dfc-b83b-7ca69301fd35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:51:02.920803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:02.920788 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c463943-87e6-4dfc-b83b-7ca69301fd35-kube-api-access-5l4lw" pod "9c463943-87e6-4dfc-b83b-7ca69301fd35" (UID: "9c463943-87e6-4dfc-b83b-7ca69301fd35"). InnerVolumeSpecName "kube-api-access-5l4lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:51:03.028513 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:03.028401 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l4lw\" (UniqueName: \"kubernetes.io/projected/9c463943-87e6-4dfc-b83b-7ca69301fd35-kube-api-access-5l4lw\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:51:03.028513 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:03.028450 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c463943-87e6-4dfc-b83b-7ca69301fd35-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:51:03.103992 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:03.103956 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:03.105633 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:03.105604 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:51:03.105775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:03.105659 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:51:03.719321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:03.719289 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" Oct 30 01:51:03.719321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:03.719313 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n" event={"ID":"9c463943-87e6-4dfc-b83b-7ca69301fd35","Type":"ContainerDied","Data":"1cc784af14e73fb1cb7b1b7953485c646602dcb043eec83742ec894d925a68d7"} Oct 30 01:51:03.719778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:03.719367 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc784af14e73fb1cb7b1b7953485c646602dcb043eec83742ec894d925a68d7" Oct 30 01:51:04.104261 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:04.104226 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:51:04.104450 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:04.104279 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:51:05.103889 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:05.103848 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:51:05.104342 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:05.103912 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:51:06.104125 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:06.104090 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:51:06.104556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:06.104160 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:51:07.103387 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:07.103350 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:51:07.103600 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:07.103424 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:51:08.104204 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:08.104170 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:51:08.104745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:08.104231 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:51:09.103957 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.103907 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:51:09.104158 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.104004 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:51:09.645110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.645073 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4"] Oct 30 01:51:09.645695 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.645677 5035 cpu_manager.go:414] "containerMap: removing container" podUID="9c463943-87e6-4dfc-b83b-7ca69301fd35" containerName="revoke-keys" Oct 30 01:51:09.645746 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.645697 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c463943-87e6-4dfc-b83b-7ca69301fd35" containerName="revoke-keys" Oct 30 01:51:09.645802 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.645792 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c463943-87e6-4dfc-b83b-7ca69301fd35" containerName="revoke-keys" Oct 30 01:51:09.649480 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.649453 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.666230 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.666203 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-719029-openshift-default-dockercfg-9jg4v\"" Oct 30 01:51:09.685430 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.685390 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4"] Oct 30 01:51:09.725383 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.725341 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.725614 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.725430 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.725614 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.725485 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkfx8\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-kube-api-access-dkfx8\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.725614 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.725606 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.725731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.725645 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.725731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.725665 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.725731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.725717 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.725822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.725789 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.725856 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.725839 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.826692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.826652 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.826692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.826693 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkfx8\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-kube-api-access-dkfx8\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.826940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.826720 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.826940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.826756 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.826940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.826782 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.826940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.826820 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.826940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.826873 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.827120 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.826946 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.827120 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.826997 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.827657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.827520 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.827657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.827549 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.827657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.827587 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.827657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.827614 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.827657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.827651 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.829390 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.829371 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.829641 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.829622 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.851067 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.851038 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.851233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.851207 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkfx8\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-kube-api-access-dkfx8\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:09.963755 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:09.963720 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:10.108616 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.108581 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:10.109003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.108983 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:10.109836 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.109818 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:51:10.290982 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:51:10.290948 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ce2c95_167f_40f3_9747_f411d0bc8ea5.slice/crio-ef395140ebde85c79b41eb1856eeb6335d8738b4d38ff3e46db4a1b6092ac943 WatchSource:0}: Error finding container ef395140ebde85c79b41eb1856eeb6335d8738b4d38ff3e46db4a1b6092ac943: Status 404 returned error can't find the container with id ef395140ebde85c79b41eb1856eeb6335d8738b4d38ff3e46db4a1b6092ac943 Oct 30 01:51:10.291686 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.291655 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4"] Oct 30 01:51:10.293492 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.293460 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:51:10.293605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.293547 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:51:10.293605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.293581 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:51:10.776194 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.771150 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" event={"ID":"a5ce2c95-167f-40f3-9747-f411d0bc8ea5","Type":"ContainerStarted","Data":"5e790a328ae4d9e08612cf0506a6a7e108236aef6ac4348eb715ab6a2310718a"} Oct 30 01:51:10.776194 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.771222 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" event={"ID":"a5ce2c95-167f-40f3-9747-f411d0bc8ea5","Type":"ContainerStarted","Data":"ef395140ebde85c79b41eb1856eeb6335d8738b4d38ff3e46db4a1b6092ac943"} Oct 30 01:51:10.826854 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.826798 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" podStartSLOduration=1.826781468 podStartE2EDuration="1.826781468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:10.823757288 +0000 UTC m=+1349.326689508" watchObservedRunningTime="2026-10-30 01:51:10.826781468 +0000 UTC m=+1349.329713920" Oct 30 01:51:10.964470 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:10.964427 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:11.966296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:11.966260 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:51:11.966854 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:11.966320 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:51:12.964187 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:12.964151 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:51:12.964385 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:12.964214 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:51:13.964344 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:13.964305 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:51:13.964883 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:13.964383 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:51:14.964176 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:14.964128 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:51:14.964400 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:14.964202 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:51:15.964231 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:15.964188 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:51:15.964431 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:15.964274 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:51:17.632193 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:51:17.632143 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:51:17.794090 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:17.794053 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:17.794441 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:17.794412 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:17.795287 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:17.795266 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 01:51:18.391669 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.391633 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn"] Oct 30 01:51:18.395616 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.395594 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.412848 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.412815 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:51:18.413033 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.412921 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:51:18.414940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.414909 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn"] Oct 30 01:51:18.552050 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.552010 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be6816c1-pgltn\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.552246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.552076 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be6816c1-pgltn\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.552246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.552209 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswvb\" (UniqueName: \"kubernetes.io/projected/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-kube-api-access-jswvb\") pod \"maas-api-revoke-sub-be6816c1-pgltn\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.653230 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.653122 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswvb\" (UniqueName: \"kubernetes.io/projected/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-kube-api-access-jswvb\") pod \"maas-api-revoke-sub-be6816c1-pgltn\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.653744 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.653242 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be6816c1-pgltn\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.653744 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.653284 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be6816c1-pgltn\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.654090 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.654066 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be6816c1-pgltn\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.655888 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.655851 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be6816c1-pgltn\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.677947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.677905 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswvb\" (UniqueName: \"kubernetes.io/projected/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-kube-api-access-jswvb\") pod \"maas-api-revoke-sub-be6816c1-pgltn\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:18.710903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:18.710858 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:19.013982 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:19.013945 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn"] Oct 30 01:51:19.017477 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:51:19.017442 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fc9164_4fa4_4468_8b0c_f301b1c91f4e.slice/crio-c9ae24efb5bb0f992619c5527ac57a5451f32216d6de8a854b209d3e9cf043a9 WatchSource:0}: Error finding container c9ae24efb5bb0f992619c5527ac57a5451f32216d6de8a854b209d3e9cf043a9: Status 404 returned error can't find the container with id c9ae24efb5bb0f992619c5527ac57a5451f32216d6de8a854b209d3e9cf043a9 Oct 30 01:51:19.856635 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:19.856598 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" event={"ID":"93fc9164-4fa4-4468-8b0c-f301b1c91f4e","Type":"ContainerStarted","Data":"b5c681775b4bc02050f7e4765105a3cb2bc4c93ac32a277b2825ab8597fe4c4c"} Oct 30 01:51:19.857153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:19.856651 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" event={"ID":"93fc9164-4fa4-4468-8b0c-f301b1c91f4e","Type":"ContainerStarted","Data":"c9ae24efb5bb0f992619c5527ac57a5451f32216d6de8a854b209d3e9cf043a9"} Oct 30 01:51:19.900898 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:19.900836 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" podStartSLOduration=1.900818942 podStartE2EDuration="1.900818942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:19.900436341 +0000 UTC m=+1358.403368562" watchObservedRunningTime="2026-10-30 01:51:19.900818942 +0000 UTC m=+1358.403751161" Oct 30 01:51:20.864896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:20.864852 5035 generic.go:356] "Generic (PLEG): container finished" podID="93fc9164-4fa4-4468-8b0c-f301b1c91f4e" containerID="b5c681775b4bc02050f7e4765105a3cb2bc4c93ac32a277b2825ab8597fe4c4c" exitCode=0 Oct 30 01:51:20.865339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:20.864902 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" event={"ID":"93fc9164-4fa4-4468-8b0c-f301b1c91f4e","Type":"ContainerDied","Data":"b5c681775b4bc02050f7e4765105a3cb2bc4c93ac32a277b2825ab8597fe4c4c"} Oct 30 01:51:22.020925 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.020892 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:22.093257 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.093222 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-kube-api-access-jswvb\" (UniqueName: \"kubernetes.io/projected/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-kube-api-access-jswvb\") pod \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " Oct 30 01:51:22.093467 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.093276 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-xks-ca\") pod \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " Oct 30 01:51:22.093467 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.093320 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-openshift-ca\") pod \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\" (UID: \"93fc9164-4fa4-4468-8b0c-f301b1c91f4e\") " Oct 30 01:51:22.093776 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.093747 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-openshift-ca" pod "93fc9164-4fa4-4468-8b0c-f301b1c91f4e" (UID: "93fc9164-4fa4-4468-8b0c-f301b1c91f4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:51:22.095785 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.095755 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-xks-ca" pod "93fc9164-4fa4-4468-8b0c-f301b1c91f4e" (UID: "93fc9164-4fa4-4468-8b0c-f301b1c91f4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:51:22.097403 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.097376 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-kube-api-access-jswvb" pod "93fc9164-4fa4-4468-8b0c-f301b1c91f4e" (UID: "93fc9164-4fa4-4468-8b0c-f301b1c91f4e"). InnerVolumeSpecName "kube-api-access-jswvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:51:22.194507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.194413 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswvb\" (UniqueName: \"kubernetes.io/projected/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-kube-api-access-jswvb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:51:22.194507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.194449 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:51:22.194507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.194459 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fc9164-4fa4-4468-8b0c-f301b1c91f4e-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:51:22.879849 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.879811 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" event={"ID":"93fc9164-4fa4-4468-8b0c-f301b1c91f4e","Type":"ContainerDied","Data":"c9ae24efb5bb0f992619c5527ac57a5451f32216d6de8a854b209d3e9cf043a9"} Oct 30 01:51:22.879849 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.879846 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn" Oct 30 01:51:22.880088 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:22.879869 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ae24efb5bb0f992619c5527ac57a5451f32216d6de8a854b209d3e9cf043a9" Oct 30 01:51:46.927199 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:46.927152 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9"] Oct 30 01:51:46.928028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:46.928004 5035 cpu_manager.go:414] "containerMap: removing container" podUID="93fc9164-4fa4-4468-8b0c-f301b1c91f4e" containerName="revoke-keys" Oct 30 01:51:46.928139 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:46.928030 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fc9164-4fa4-4468-8b0c-f301b1c91f4e" containerName="revoke-keys" Oct 30 01:51:46.928231 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:46.928218 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fc9164-4fa4-4468-8b0c-f301b1c91f4e" containerName="revoke-keys" Oct 30 01:51:46.933042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:46.933015 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:46.964747 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:46.964695 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9"] Oct 30 01:51:47.095227 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.095190 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.095227 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.095234 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.095455 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.095366 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.095455 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.095404 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.095543 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.095467 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.095583 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.095545 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.095583 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.095577 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.095657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.095610 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.095657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.095643 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn87s\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-kube-api-access-mn87s\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197005 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.196905 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197005 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.196954 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197005 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.196993 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197298 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197058 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197298 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197097 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197298 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197130 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197298 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197149 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn87s\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-kube-api-access-mn87s\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197298 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197227 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197298 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197262 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197487 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197578 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197723 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197626 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197833 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197809 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.197926 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.197904 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.199606 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.199582 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.199836 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.199819 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.223738 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.223705 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.251889 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.251825 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn87s\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-kube-api-access-mn87s\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.547916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.547874 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:47.643835 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:51:47.643795 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:51:47.871189 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.871160 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9"] Oct 30 01:51:47.872751 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:51:47.872719 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b86e2b2_8b3d_4f5d_aea6_c19796c03c13.slice/crio-343b155b926e353a6c19e1ecdef2d8c0f0da41d0f1b8523decc97be0ffd64005 WatchSource:0}: Error finding container 343b155b926e353a6c19e1ecdef2d8c0f0da41d0f1b8523decc97be0ffd64005: Status 404 returned error can't find the container with id 343b155b926e353a6c19e1ecdef2d8c0f0da41d0f1b8523decc97be0ffd64005 Oct 30 01:51:47.875199 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.875166 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:51:47.875297 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.875235 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:51:47.875297 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:47.875264 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:51:48.055857 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:48.055821 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" event={"ID":"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13","Type":"ContainerStarted","Data":"6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373"} Oct 30 01:51:48.056320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:48.055884 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" event={"ID":"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13","Type":"ContainerStarted","Data":"343b155b926e353a6c19e1ecdef2d8c0f0da41d0f1b8523decc97be0ffd64005"} Oct 30 01:51:48.104470 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:48.104394 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" podStartSLOduration=2.104373262 podStartE2EDuration="2.104373262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:48.103880252 +0000 UTC m=+1386.606812471" watchObservedRunningTime="2026-10-30 01:51:48.104373262 +0000 UTC m=+1386.607305483" Oct 30 01:51:48.548295 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:48.548248 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:48.550287 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:48.550257 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:51:48.550432 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:48.550323 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:51:49.548582 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:49.548512 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:51:49.549043 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:49.548598 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:51:50.548642 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:50.548602 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:51:50.549081 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:50.548665 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:51:51.548508 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:51.548459 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:51:51.548734 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:51.548520 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:51:52.549193 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:52.549146 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:51:52.549760 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:52.549215 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:51:53.549050 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:53.549011 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:51:53.549276 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:53.549098 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:51:54.548569 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:54.548489 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:51:54.548813 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:54.548616 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:51:55.553675 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:55.553638 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:55.554101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:55.554027 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:55.554990 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:55.554966 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:51:58.367346 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.367303 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv"] Oct 30 01:51:58.372689 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.372662 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.393108 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.393075 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv"] Oct 30 01:51:58.402907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.402861 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"kube-root-ca.crt\"" Oct 30 01:51:58.402907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.402861 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"e2e-conflict-b-e338a073-kserve-self-signed-certs\"" Oct 30 01:51:58.403117 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.402932 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"default-dockercfg-clrdr\"" Oct 30 01:51:58.403117 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.402973 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"openshift-service-ca.crt\"" Oct 30 01:51:58.455088 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.455052 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-model-cache\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.455088 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.455098 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tmp-dir\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.455325 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.455184 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-home\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.455325 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.455235 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-dshm\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.455393 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.455334 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tls-certs\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.455393 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.455356 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t764\" (UniqueName: \"kubernetes.io/projected/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-kube-api-access-4t764\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.557105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.557056 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-model-cache\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.557300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.557115 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tmp-dir\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.557300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.557156 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-home\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.557300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.557179 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-dshm\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.557300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.557295 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tls-certs\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.557492 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.557329 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t764\" (UniqueName: \"kubernetes.io/projected/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-kube-api-access-4t764\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.557638 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.557617 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-model-cache\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.557714 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.557665 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tmp-dir\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.557782 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.557751 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-home\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.559785 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.559753 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-dshm\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.560003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.559986 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tls-certs\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.597655 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.597580 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t764\" (UniqueName: \"kubernetes.io/projected/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-kube-api-access-4t764\") pod \"e2e-conflict-b-e338a073-kserve-657b654c49-lrljv\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.691008 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.690913 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:51:58.996584 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:58.996554 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv"] Oct 30 01:51:58.998205 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:51:58.998173 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod925ae2a6_20e3_4737_9e1b_cadcc7850ffa.slice/crio-6bd53f6702b2b1772f4b79a263da8f117d66a21f050af2da0de47599848a4802 WatchSource:0}: Error finding container 6bd53f6702b2b1772f4b79a263da8f117d66a21f050af2da0de47599848a4802: Status 404 returned error can't find the container with id 6bd53f6702b2b1772f4b79a263da8f117d66a21f050af2da0de47599848a4802 Oct 30 01:51:59.161048 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:59.161008 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" event={"ID":"925ae2a6-20e3-4737-9e1b-cadcc7850ffa","Type":"ContainerStarted","Data":"7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d"} Oct 30 01:51:59.161232 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:59.161065 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" event={"ID":"925ae2a6-20e3-4737-9e1b-cadcc7850ffa","Type":"ContainerStarted","Data":"6bd53f6702b2b1772f4b79a263da8f117d66a21f050af2da0de47599848a4802"} Oct 30 01:51:59.207200 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:59.207140 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" podStartSLOduration=1.207123606 podStartE2EDuration="1.207123606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:59.205206662 +0000 UTC m=+1397.708138882" watchObservedRunningTime="2026-10-30 01:51:59.207123606 +0000 UTC m=+1397.710055826" Oct 30 01:51:59.691289 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:51:59.691243 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:52:02.706222 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.706177 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn"] Oct 30 01:52:02.710796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.710769 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.729170 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.729136 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn"] Oct 30 01:52:02.738229 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.738191 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-719029\"/\"distinct-w2-719029-kserve-self-signed-certs\"" Oct 30 01:52:02.808344 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.808296 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-dshm\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.808344 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.808345 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-home\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.808684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.808421 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9hbc\" (UniqueName: \"kubernetes.io/projected/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-kube-api-access-d9hbc\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.808684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.808544 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-model-cache\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.808684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.808576 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tmp-dir\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.808803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.808687 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tls-certs\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.916466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.916418 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tls-certs\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.916680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.916506 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-dshm\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.916680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.916554 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-home\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.916680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.916598 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9hbc\" (UniqueName: \"kubernetes.io/projected/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-kube-api-access-d9hbc\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.916792 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.916687 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-model-cache\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.916792 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.916717 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tmp-dir\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.917211 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.917190 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tmp-dir\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.918443 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.918411 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-home\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.918739 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.918717 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-model-cache\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.929469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.929435 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tls-certs\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.936961 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.936911 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-dshm\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:02.943232 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:02.943196 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9hbc\" (UniqueName: \"kubernetes.io/projected/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-kube-api-access-d9hbc\") pod \"distinct-w2-719029-kserve-7b54b578c5-mvkpn\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:03.025227 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:03.025126 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:03.334016 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:03.333978 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn"] Oct 30 01:52:03.335211 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:52:03.335182 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbbca3d1_d147_4558_aa31_9f7f6fd0fcd1.slice/crio-80abb30f24d3bcae2ac87611226334c5d7af723514175a56d5efcdb0fd8dc65d WatchSource:0}: Error finding container 80abb30f24d3bcae2ac87611226334c5d7af723514175a56d5efcdb0fd8dc65d: Status 404 returned error can't find the container with id 80abb30f24d3bcae2ac87611226334c5d7af723514175a56d5efcdb0fd8dc65d Oct 30 01:52:04.194650 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:04.194605 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" event={"ID":"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1","Type":"ContainerStarted","Data":"6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8"} Oct 30 01:52:04.195140 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:04.194664 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" event={"ID":"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1","Type":"ContainerStarted","Data":"80abb30f24d3bcae2ac87611226334c5d7af723514175a56d5efcdb0fd8dc65d"} Oct 30 01:52:04.240632 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:04.240566 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podStartSLOduration=2.240546517 podStartE2EDuration="2.240546517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:04.238210975 +0000 UTC m=+1402.741143195" watchObservedRunningTime="2026-10-30 01:52:04.240546517 +0000 UTC m=+1402.743478737" Oct 30 01:52:05.025812 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:05.025775 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:08.211851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.211805 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns"] Oct 30 01:52:08.229717 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.229305 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.234950 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.234915 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns"] Oct 30 01:52:08.272506 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.272472 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"kube-root-ca.crt\"" Oct 30 01:52:08.272720 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.272702 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"openshift-service-ca.crt\"" Oct 30 01:52:08.272803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.272784 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"distinct-2-w0-c40a89-kserve-self-signed-certs\"" Oct 30 01:52:08.288134 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.288092 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-home\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.288306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.288161 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tmp-dir\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.288306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.288256 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-model-cache\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.288380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.288328 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tls-certs\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.288429 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.288416 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njgtq\" (UniqueName: \"kubernetes.io/projected/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-kube-api-access-njgtq\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.288468 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.288452 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-dshm\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.362416 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.362383 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"default-dockercfg-rp6jt\"" Oct 30 01:52:08.389860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.389824 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tmp-dir\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.390209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.389879 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-model-cache\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.390209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.389936 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tls-certs\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.390209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.390021 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njgtq\" (UniqueName: \"kubernetes.io/projected/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-kube-api-access-njgtq\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.390209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.390071 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-dshm\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.390457 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.390275 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-home\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.390457 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.390387 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tmp-dir\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.390618 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.390455 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-model-cache\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.390684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.390663 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-home\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.392648 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.392617 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-dshm\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.392842 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.392823 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tls-certs\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.415331 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.415297 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njgtq\" (UniqueName: \"kubernetes.io/projected/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-kube-api-access-njgtq\") pod \"distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.546898 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.546802 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:08.691874 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.691824 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:52:08.705108 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.705068 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:52:08.853389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:08.853361 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns"] Oct 30 01:52:08.857589 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:52:08.855156 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ab24a6c_d9c8_4c42_b29b_24a7e2a7bfe0.slice/crio-7ee5577868e9e8c03c93b9c0793535d3d643b3963c8e560b0df0a79ef284c8f9 WatchSource:0}: Error finding container 7ee5577868e9e8c03c93b9c0793535d3d643b3963c8e560b0df0a79ef284c8f9: Status 404 returned error can't find the container with id 7ee5577868e9e8c03c93b9c0793535d3d643b3963c8e560b0df0a79ef284c8f9 Oct 30 01:52:09.239359 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.239315 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" event={"ID":"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0","Type":"ContainerStarted","Data":"865f4e8defcda3400581281601bb9de915e145ae474717286a46ab0a9e70e5e3"} Oct 30 01:52:09.239901 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.239373 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" event={"ID":"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0","Type":"ContainerStarted","Data":"7ee5577868e9e8c03c93b9c0793535d3d643b3963c8e560b0df0a79ef284c8f9"} Oct 30 01:52:09.251897 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.251855 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:52:09.287925 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.287851 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podStartSLOduration=1.287835098 podStartE2EDuration="1.287835098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:09.283646839 +0000 UTC m=+1407.786579058" watchObservedRunningTime="2026-10-30 01:52:09.287835098 +0000 UTC m=+1407.790767317" Oct 30 01:52:09.547662 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.547564 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:09.824706 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.824610 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7"] Oct 30 01:52:09.829226 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.829192 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:09.854241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.854205 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7"] Oct 30 01:52:09.907606 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.907565 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5083785-8e42-4372-8d43-49cc6edceda4-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:09.907803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.907615 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhhg4\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-kube-api-access-lhhg4\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:09.907803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.907678 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:09.907803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.907696 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5083785-8e42-4372-8d43-49cc6edceda4-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:09.907916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.907793 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:09.907916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.907836 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:09.907991 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.907951 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:09.908099 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.908080 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:09.908152 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:09.908139 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009028 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.008976 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009235 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009136 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009235 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009184 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009240 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5083785-8e42-4372-8d43-49cc6edceda4-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009278 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhhg4\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-kube-api-access-lhhg4\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009307 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009334 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5083785-8e42-4372-8d43-49cc6edceda4-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009384 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009420 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009660 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009582 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009872 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009842 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.009965 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009893 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.010029 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.009977 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.010202 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.010176 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5083785-8e42-4372-8d43-49cc6edceda4-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.011614 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.011595 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.012464 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.012447 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5083785-8e42-4372-8d43-49cc6edceda4-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.035026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.034981 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.035183 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.034991 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhhg4\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-kube-api-access-lhhg4\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.151392 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.151293 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:10.469697 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.469665 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7"] Oct 30 01:52:10.471347 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:52:10.471302 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5083785_8e42_4372_8d43_49cc6edceda4.slice/crio-c5c1b9c1e3fa31fd259ae90894fc6242dc753b4c8ba1ac0e4ae6c46053313612 WatchSource:0}: Error finding container c5c1b9c1e3fa31fd259ae90894fc6242dc753b4c8ba1ac0e4ae6c46053313612: Status 404 returned error can't find the container with id c5c1b9c1e3fa31fd259ae90894fc6242dc753b4c8ba1ac0e4ae6c46053313612 Oct 30 01:52:10.474261 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.474220 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:10.474359 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.474314 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:10.474397 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:10.474366 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:11.256380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:11.256343 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" event={"ID":"d5083785-8e42-4372-8d43-49cc6edceda4","Type":"ContainerStarted","Data":"44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d"} Oct 30 01:52:11.256655 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:11.256404 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" event={"ID":"d5083785-8e42-4372-8d43-49cc6edceda4","Type":"ContainerStarted","Data":"c5c1b9c1e3fa31fd259ae90894fc6242dc753b4c8ba1ac0e4ae6c46053313612"} Oct 30 01:52:11.360766 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:11.360706 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" podStartSLOduration=2.36068507 podStartE2EDuration="2.36068507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:11.357930946 +0000 UTC m=+1409.860863169" watchObservedRunningTime="2026-10-30 01:52:11.36068507 +0000 UTC m=+1409.863617289" Oct 30 01:52:12.152125 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:12.152091 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:12.153835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:12.153807 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:52:12.153976 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:12.153858 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:52:13.026404 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:13.026233 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:13.039742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:13.039710 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:13.152224 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:13.152191 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:52:13.152713 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:13.152250 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:52:13.305436 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:13.305346 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:52:14.152280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:14.152237 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:52:14.152756 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:14.152304 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:52:15.152286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:15.152241 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:52:15.152699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:15.152316 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:52:16.152075 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:16.152030 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:52:16.152439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:16.152400 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:52:17.152019 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:17.151979 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:52:17.152214 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:17.152067 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:52:17.630659 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:52:17.630613 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:52:18.324184 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:18.324151 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:18.326007 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:18.325832 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:18.328034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:18.328002 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 01:52:18.547428 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:18.547389 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:18.560585 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:18.560549 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:19.378372 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:19.378333 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:52:32.071849 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.071809 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst"] Oct 30 01:52:32.076086 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.076055 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.099597 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.099559 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst"] Oct 30 01:52:32.142756 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.142701 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.142970 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.142875 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71474faf-7414-4bbb-8d8d-592e30db5efc-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.142970 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.142927 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-kube-api-access-qfnzr\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.143114 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.143073 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.143114 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.143104 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.143233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.143140 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.143233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.143171 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.143233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.143195 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.143385 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.143370 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244324 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244276 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244387 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71474faf-7414-4bbb-8d8d-592e30db5efc-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244427 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-kube-api-access-qfnzr\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244498 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244539 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244602 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244638 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244762 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244978 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244827 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.244978 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.244936 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.245086 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.245059 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.245142 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.245130 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.245320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.245298 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.245399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.245324 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71474faf-7414-4bbb-8d8d-592e30db5efc-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.246959 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.246924 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.247217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.247196 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.269380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.269345 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-kube-api-access-qfnzr\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.269553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.269346 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.394145 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.394041 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:32.725542 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.725480 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst"] Oct 30 01:52:32.725988 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:52:32.725966 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71474faf_7414_4bbb_8d8d_592e30db5efc.slice/crio-de07b3622ba41ce23cf5aabc9a8d985682848cef667e4bc980c3cd7229a20b6c WatchSource:0}: Error finding container de07b3622ba41ce23cf5aabc9a8d985682848cef667e4bc980c3cd7229a20b6c: Status 404 returned error can't find the container with id de07b3622ba41ce23cf5aabc9a8d985682848cef667e4bc980c3cd7229a20b6c Oct 30 01:52:32.728623 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.728585 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:32.728743 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.728675 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:32.728743 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:32.728724 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:33.489554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:33.480589 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" event={"ID":"71474faf-7414-4bbb-8d8d-592e30db5efc","Type":"ContainerStarted","Data":"67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f"} Oct 30 01:52:33.489554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:33.480667 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" event={"ID":"71474faf-7414-4bbb-8d8d-592e30db5efc","Type":"ContainerStarted","Data":"de07b3622ba41ce23cf5aabc9a8d985682848cef667e4bc980c3cd7229a20b6c"} Oct 30 01:52:34.394304 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:34.394254 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:34.396074 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:34.396043 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:52:34.396207 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:34.396142 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:52:35.139944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:35.139595 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" podStartSLOduration=3.139574899 podStartE2EDuration="3.139574899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:33.554827015 +0000 UTC m=+1432.057759235" watchObservedRunningTime="2026-10-30 01:52:35.139574899 +0000 UTC m=+1433.642507120" Oct 30 01:52:35.142490 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:35.142458 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv"] Oct 30 01:52:35.394814 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:35.394671 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:52:35.394814 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:35.394756 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:52:36.394654 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:36.394612 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:52:36.395122 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:36.394698 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:52:37.399556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:37.398915 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:52:37.399556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:37.398983 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:52:38.395242 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:38.395202 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:52:38.395552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:38.395503 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:52:39.395082 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.395040 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:52:39.395597 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.395114 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:52:39.944944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.944895 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld"] Oct 30 01:52:39.951480 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.951456 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:39.975739 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.975707 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld"] Oct 30 01:52:39.996278 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.996204 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:39.996496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.996426 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:39.996496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.996490 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07e40cb-209d-47ee-9324-cd445c485cb6-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:39.996665 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.996574 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:39.996665 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.996630 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lcg\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-kube-api-access-k7lcg\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:39.996778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.996722 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:39.997310 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.996841 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:39.997310 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.997033 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:39.997310 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:39.997084 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.098822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.098771 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.098822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.098825 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.098857 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.098905 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.098978 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.099007 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07e40cb-209d-47ee-9324-cd445c485cb6-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.099063 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.099103 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lcg\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-kube-api-access-k7lcg\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099463 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.099161 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099520 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.099451 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099666 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.099551 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099666 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.099641 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.099837 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.099806 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.100120 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.099882 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07e40cb-209d-47ee-9324-cd445c485cb6-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.101866 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.101834 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.102212 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.102184 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.132701 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.132660 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lcg\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-kube-api-access-k7lcg\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.133167 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.133143 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.272265 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.272227 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:40.400073 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.400020 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:40.401084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.401057 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:40.401895 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.401866 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 01:52:40.603981 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.603940 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld"] Oct 30 01:52:40.605864 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:52:40.605834 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07e40cb_209d_47ee_9324_cd445c485cb6.slice/crio-5a8778230750b6e15c24d896c866c2cceb3e707e68bfb09f5ac4c81956810534 WatchSource:0}: Error finding container 5a8778230750b6e15c24d896c866c2cceb3e707e68bfb09f5ac4c81956810534: Status 404 returned error can't find the container with id 5a8778230750b6e15c24d896c866c2cceb3e707e68bfb09f5ac4c81956810534 Oct 30 01:52:40.608680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.608645 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:40.608836 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.608716 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:40.608836 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:40.608749 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:41.549307 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:41.549264 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" event={"ID":"d07e40cb-209d-47ee-9324-cd445c485cb6","Type":"ContainerStarted","Data":"a7e5bd42f6964847b701bcd96fe497a8cfb817f7789ad6787d6601f5048b41b5"} Oct 30 01:52:41.549867 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:41.549355 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" event={"ID":"d07e40cb-209d-47ee-9324-cd445c485cb6","Type":"ContainerStarted","Data":"5a8778230750b6e15c24d896c866c2cceb3e707e68bfb09f5ac4c81956810534"} Oct 30 01:52:41.612444 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:41.612388 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" podStartSLOduration=2.612369895 podStartE2EDuration="2.612369895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:41.611765285 +0000 UTC m=+1440.114697502" watchObservedRunningTime="2026-10-30 01:52:41.612369895 +0000 UTC m=+1440.115302115" Oct 30 01:52:42.277003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:42.276967 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:42.284791 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:42.284743 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:52:42.285119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:42.285070 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:52:43.286554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:43.284037 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:52:43.286554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:43.284104 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:52:44.274097 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:44.274044 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:52:44.274431 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:44.274400 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:52:45.274041 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:45.273997 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:52:45.274715 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:45.274674 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:52:46.282553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:46.273998 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:52:46.282553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:46.274071 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:52:47.273492 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:47.273446 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:52:47.273706 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:47.273519 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:52:47.635404 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:52:47.635292 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:52:48.278287 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:48.278246 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:48.278822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:48.278799 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:48.279803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:48.279785 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 01:52:50.204555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:50.198862 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" podUID="925ae2a6-20e3-4737-9e1b-cadcc7850ffa" containerName="main" containerID="cri-o://7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d" gracePeriod=45 Oct 30 01:52:50.254588 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:50.244990 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" podUID="925ae2a6-20e3-4737-9e1b-cadcc7850ffa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused" Oct 30 01:52:51.040628 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.040571 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:52:51.063892 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.063843 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg"] Oct 30 01:52:51.088209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.088176 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.210554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.202176 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg"] Oct 30 01:52:51.286007 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.285952 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" podUID="925ae2a6-20e3-4737-9e1b-cadcc7850ffa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused" Oct 30 01:52:51.286384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.286358 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh"] Oct 30 01:52:51.294494 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.292960 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:52:51.295070 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.295022 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh"] Oct 30 01:52:51.295274 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.293386 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.312332 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.312291 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.312656 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.312631 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.312837 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.312820 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.312950 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.312934 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.313080 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.313066 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.313209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.313193 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.313382 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.313367 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.313708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.313522 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gx8g\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-kube-api-access-2gx8g\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.314022 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.314000 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.418136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.417378 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.418136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.417844 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.418136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.417991 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.418136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.418076 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.418593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.418159 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2r22\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-kube-api-access-n2r22\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.418593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.418191 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.418593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.418246 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.419114 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.419236 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.419287 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.419370 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.419571 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.420110 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.420177 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.420253 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gx8g\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-kube-api-access-2gx8g\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.420607 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.420278 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.420984 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.421068 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.421381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.421119 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.422187 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.421575 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.422187 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.421826 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.423586 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.422954 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.437379 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.437296 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.443426 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.443003 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.444969 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.444945 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.445380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.445362 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gx8g\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-kube-api-access-2gx8g\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.522944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.522905 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.522956 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.522992 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2r22\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-kube-api-access-n2r22\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523022 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523061 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523098 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523442 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523192 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523442 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523344 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523442 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523408 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523603 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523450 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523685 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523659 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.523814 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523765 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.524002 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.523980 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.524257 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.524239 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.525699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.525663 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.525955 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.525938 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.547545 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.547460 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.547545 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.547498 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2r22\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-kube-api-access-n2r22\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.574514 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.574482 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:52:51.589981 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.589942 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:51.626603 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.626571 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:51.687360 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.687184 5035 generic.go:356] "Generic (PLEG): container finished" podID="925ae2a6-20e3-4737-9e1b-cadcc7850ffa" containerID="7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d" exitCode=0 Oct 30 01:52:51.687994 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.687571 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" event={"ID":"925ae2a6-20e3-4737-9e1b-cadcc7850ffa","Type":"ContainerDied","Data":"7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d"} Oct 30 01:52:51.687994 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.687612 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" event={"ID":"925ae2a6-20e3-4737-9e1b-cadcc7850ffa","Type":"ContainerDied","Data":"6bd53f6702b2b1772f4b79a263da8f117d66a21f050af2da0de47599848a4802"} Oct 30 01:52:51.687994 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.687630 5035 scope.go:122] "RemoveContainer" containerID="7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d" Oct 30 01:52:51.687994 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.687766 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv" Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.724991 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-model-cache\") pod \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.725101 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-home\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-home\") pod \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.725212 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tmp-dir\") pod \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.725257 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-kube-api-access-4t764\" (UniqueName: \"kubernetes.io/projected/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-kube-api-access-4t764\") pod \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.725381 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tls-certs\" (UniqueName: \"kubernetes.io/secret/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tls-certs\") pod \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.725480 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-dshm\") pod \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\" (UID: \"925ae2a6-20e3-4737-9e1b-cadcc7850ffa\") " Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.727898 5035 scope.go:122] "RemoveContainer" containerID="7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d" Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.730168 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tls-certs" pod "925ae2a6-20e3-4737-9e1b-cadcc7850ffa" (UID: "925ae2a6-20e3-4737-9e1b-cadcc7850ffa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.730354 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-home" pod "925ae2a6-20e3-4737-9e1b-cadcc7850ffa" (UID: "925ae2a6-20e3-4737-9e1b-cadcc7850ffa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:52:51.730396 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d\": container with ID starting with 7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d not found: ID does not exist" containerID="7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d" Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.730436 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d"} err="failed to get container status \"7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d\": rpc error: code = NotFound desc = could not find container \"7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d\": container with ID starting with 7c5759713dc33687a3409e3eb1115fa5fb25a39e96169a671761a0e5f6023b4d not found: ID does not exist" Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.730488 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-model-cache" pod "925ae2a6-20e3-4737-9e1b-cadcc7850ffa" (UID: "925ae2a6-20e3-4737-9e1b-cadcc7850ffa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:51.731306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.731269 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tmp-dir" pod "925ae2a6-20e3-4737-9e1b-cadcc7850ffa" (UID: "925ae2a6-20e3-4737-9e1b-cadcc7850ffa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:51.733332 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.733291 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-kube-api-access-4t764" pod "925ae2a6-20e3-4737-9e1b-cadcc7850ffa" (UID: "925ae2a6-20e3-4737-9e1b-cadcc7850ffa"). InnerVolumeSpecName "kube-api-access-4t764". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:51.733553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.733453 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:52:51.733553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.733487 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:52:51.733553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.733502 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:52:51.733553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.733515 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:52:51.734601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.734569 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-dshm" pod "925ae2a6-20e3-4737-9e1b-cadcc7850ffa" (UID: "925ae2a6-20e3-4737-9e1b-cadcc7850ffa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:51.835338 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.835243 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t764\" (UniqueName: \"kubernetes.io/projected/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-kube-api-access-4t764\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:52:51.835338 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.835277 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/925ae2a6-20e3-4737-9e1b-cadcc7850ffa-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:52:51.931060 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.931021 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg"] Oct 30 01:52:51.932320 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:52:51.932289 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e47036_a11e_4f4f_90fd_c36e6d7198d2.slice/crio-94c00015a045c174ff2da43ba68b579f0543b7016f8768f4bf2ff649188c3396 WatchSource:0}: Error finding container 94c00015a045c174ff2da43ba68b579f0543b7016f8768f4bf2ff649188c3396: Status 404 returned error can't find the container with id 94c00015a045c174ff2da43ba68b579f0543b7016f8768f4bf2ff649188c3396 Oct 30 01:52:51.934879 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.934844 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:51.934956 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.934914 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:51.934956 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.934944 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:51.970278 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.970244 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh"] Oct 30 01:52:51.975952 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:52:51.975909 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a5121f_31c1_4f5a_a278_9cf5146b78ec.slice/crio-38a6738633d677717850327920e7d82a7230a476c0ef00fd6df204f8789d7cbc WatchSource:0}: Error finding container 38a6738633d677717850327920e7d82a7230a476c0ef00fd6df204f8789d7cbc: Status 404 returned error can't find the container with id 38a6738633d677717850327920e7d82a7230a476c0ef00fd6df204f8789d7cbc Oct 30 01:52:51.978906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.978866 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:51.979003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.978969 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:51.979061 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:51.979044 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:52:52.037169 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:52.037092 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv"] Oct 30 01:52:52.043135 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:52.043099 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-b-e338a073-kserve-657b654c49-lrljv"] Oct 30 01:52:52.729082 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:52.728836 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" event={"ID":"b6e47036-a11e-4f4f-90fd-c36e6d7198d2","Type":"ContainerStarted","Data":"16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037"} Oct 30 01:52:52.729082 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:52.728899 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" event={"ID":"b6e47036-a11e-4f4f-90fd-c36e6d7198d2","Type":"ContainerStarted","Data":"94c00015a045c174ff2da43ba68b579f0543b7016f8768f4bf2ff649188c3396"} Oct 30 01:52:52.731789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:52.731756 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" event={"ID":"36a5121f-31c1-4f5a-a278-9cf5146b78ec","Type":"ContainerStarted","Data":"9f85789798b64e51aaaf2f4a3016ba88662c41d70cd9c555bdcd08fd1d177aba"} Oct 30 01:52:52.731881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:52.731815 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" event={"ID":"36a5121f-31c1-4f5a-a278-9cf5146b78ec","Type":"ContainerStarted","Data":"38a6738633d677717850327920e7d82a7230a476c0ef00fd6df204f8789d7cbc"} Oct 30 01:52:52.864313 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:52.864251 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podStartSLOduration=1.8642361109999999 podStartE2EDuration="1.864236111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:52.816356453 +0000 UTC m=+1451.319288673" watchObservedRunningTime="2026-10-30 01:52:52.864236111 +0000 UTC m=+1451.367168330" Oct 30 01:52:53.598554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:53.597980 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:53.598554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:53.598186 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:52:53.598554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:53.598235 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 01:52:53.627778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:53.627736 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:53.629380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:53.629355 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:52:53.629440 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:53.629406 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:52:53.644106 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:53.644071 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925ae2a6-20e3-4737-9e1b-cadcc7850ffa" path="/var/lib/kubelet/pods/925ae2a6-20e3-4737-9e1b-cadcc7850ffa/volumes" Oct 30 01:52:54.598054 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:54.598011 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:52:54.598742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:54.598703 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 01:52:54.628315 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:54.628273 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:52:54.628606 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:54.628574 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:52:55.592064 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:55.592019 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:52:55.592411 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:55.592376 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 01:52:55.627770 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:55.627732 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:52:55.628179 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:55.627795 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:52:56.598554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:56.597998 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:52:56.598554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:56.598071 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 01:52:56.627169 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:56.627108 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:52:56.627540 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:56.627477 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:52:57.591058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:57.591014 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:52:57.591700 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:57.591664 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 01:52:57.627254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:57.627215 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:52:57.627492 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:57.627471 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:52:58.590767 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:58.590715 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:52:58.590970 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:58.590774 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 01:52:58.627254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:58.627207 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:52:58.627753 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:58.627285 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:52:59.727423 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:59.727384 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:59.728834 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:59.728787 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:59.729656 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:59.729631 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:52:59.866701 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:59.863641 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:59.866701 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:59.865291 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:52:59.872326 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:59.872254 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podStartSLOduration=8.872229385 podStartE2EDuration="8.872229385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:52.865779188 +0000 UTC m=+1451.368711421" watchObservedRunningTime="2026-10-30 01:52:59.872229385 +0000 UTC m=+1458.375161606" Oct 30 01:52:59.876430 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:52:59.876351 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:53:01.415555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:01.410390 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:01.432914 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:01.432863 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:01.433657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:01.433209 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:53:01.468720 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:01.468662 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:06.342070 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.342032 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr"] Oct 30 01:53:06.342701 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.342686 5035 cpu_manager.go:414] "containerMap: removing container" podUID="925ae2a6-20e3-4737-9e1b-cadcc7850ffa" containerName="main" Oct 30 01:53:06.342750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.342702 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925ae2a6-20e3-4737-9e1b-cadcc7850ffa" containerName="main" Oct 30 01:53:06.342816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.342800 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="925ae2a6-20e3-4737-9e1b-cadcc7850ffa" containerName="main" Oct 30 01:53:06.346767 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.346744 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.366002 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.365971 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr"] Oct 30 01:53:06.373744 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.373698 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:06.373943 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.373902 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:06.455671 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.455631 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82948b86-bsrwr\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.455857 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.455784 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82948b86-bsrwr\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.455919 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.455858 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsckk\" (UniqueName: \"kubernetes.io/projected/7bd8ad70-8874-49aa-970f-7f58401c2efe-kube-api-access-vsckk\") pod \"maas-api-revoke-sub-82948b86-bsrwr\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.557734 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.557694 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82948b86-bsrwr\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.557895 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.557782 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82948b86-bsrwr\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.557895 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.557841 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsckk\" (UniqueName: \"kubernetes.io/projected/7bd8ad70-8874-49aa-970f-7f58401c2efe-kube-api-access-vsckk\") pod \"maas-api-revoke-sub-82948b86-bsrwr\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.558562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.558520 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82948b86-bsrwr\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.560312 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.560284 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82948b86-bsrwr\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.582498 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.582468 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsckk\" (UniqueName: \"kubernetes.io/projected/7bd8ad70-8874-49aa-970f-7f58401c2efe-kube-api-access-vsckk\") pod \"maas-api-revoke-sub-82948b86-bsrwr\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.661326 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.661242 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:06.970326 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.970292 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr"] Oct 30 01:53:06.971846 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:53:06.971813 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd8ad70_8874_49aa_970f_7f58401c2efe.slice/crio-cc74d7b4ba933a024f1f3881684cf8e7084bb30aed195994a53b93d14cee6c8c WatchSource:0}: Error finding container cc74d7b4ba933a024f1f3881684cf8e7084bb30aed195994a53b93d14cee6c8c: Status 404 returned error can't find the container with id cc74d7b4ba933a024f1f3881684cf8e7084bb30aed195994a53b93d14cee6c8c Oct 30 01:53:06.986906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:06.986867 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" event={"ID":"7bd8ad70-8874-49aa-970f-7f58401c2efe","Type":"ContainerStarted","Data":"cc74d7b4ba933a024f1f3881684cf8e7084bb30aed195994a53b93d14cee6c8c"} Oct 30 01:53:07.995097 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:07.995056 5035 generic.go:356] "Generic (PLEG): container finished" podID="7bd8ad70-8874-49aa-970f-7f58401c2efe" containerID="a9d95ad03efafca066a8cd853537a85d8dc103ee53f867503a0c48add892c22a" exitCode=0 Oct 30 01:53:07.995513 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:07.995139 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" event={"ID":"7bd8ad70-8874-49aa-970f-7f58401c2efe","Type":"ContainerDied","Data":"a9d95ad03efafca066a8cd853537a85d8dc103ee53f867503a0c48add892c22a"} Oct 30 01:53:09.195316 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.195285 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:09.292614 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.292476 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-xks-ca\") pod \"7bd8ad70-8874-49aa-970f-7f58401c2efe\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " Oct 30 01:53:09.292810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.292663 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd8ad70-8874-49aa-970f-7f58401c2efe-kube-api-access-vsckk\" (UniqueName: \"kubernetes.io/projected/7bd8ad70-8874-49aa-970f-7f58401c2efe-kube-api-access-vsckk\") pod \"7bd8ad70-8874-49aa-970f-7f58401c2efe\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " Oct 30 01:53:09.292810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.292724 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-openshift-ca\") pod \"7bd8ad70-8874-49aa-970f-7f58401c2efe\" (UID: \"7bd8ad70-8874-49aa-970f-7f58401c2efe\") " Oct 30 01:53:09.293121 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.293087 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-openshift-ca" pod "7bd8ad70-8874-49aa-970f-7f58401c2efe" (UID: "7bd8ad70-8874-49aa-970f-7f58401c2efe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:09.293340 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.293317 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.294917 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.294884 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-xks-ca" pod "7bd8ad70-8874-49aa-970f-7f58401c2efe" (UID: "7bd8ad70-8874-49aa-970f-7f58401c2efe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:09.295040 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.294963 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd8ad70-8874-49aa-970f-7f58401c2efe-kube-api-access-vsckk" pod "7bd8ad70-8874-49aa-970f-7f58401c2efe" (UID: "7bd8ad70-8874-49aa-970f-7f58401c2efe"). InnerVolumeSpecName "kube-api-access-vsckk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:09.395253 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.395211 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsckk\" (UniqueName: \"kubernetes.io/projected/7bd8ad70-8874-49aa-970f-7f58401c2efe-kube-api-access-vsckk\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.395253 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.395244 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd8ad70-8874-49aa-970f-7f58401c2efe-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.621749 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.621641 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb"] Oct 30 01:53:09.622926 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.622891 5035 cpu_manager.go:414] "containerMap: removing container" podUID="7bd8ad70-8874-49aa-970f-7f58401c2efe" containerName="revoke-keys" Oct 30 01:53:09.622926 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.622923 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd8ad70-8874-49aa-970f-7f58401c2efe" containerName="revoke-keys" Oct 30 01:53:09.623141 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.623128 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd8ad70-8874-49aa-970f-7f58401c2efe" containerName="revoke-keys" Oct 30 01:53:09.627338 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.627315 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.648244 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.648212 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb"] Oct 30 01:53:09.652518 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.652483 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"distinct-w4-b93d51-kserve-self-signed-certs\"" Oct 30 01:53:09.800107 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.800056 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df753c92-98ff-4ceb-8468-6766ded63515-tls-certs\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.800330 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.800189 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-model-cache\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.800330 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.800250 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-tmp-dir\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.800455 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.800387 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfznq\" (UniqueName: \"kubernetes.io/projected/df753c92-98ff-4ceb-8468-6766ded63515-kube-api-access-cfznq\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.800520 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.800498 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-dshm\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.800739 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.800722 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-home\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.902099 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.902005 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-home\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.902284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.902117 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df753c92-98ff-4ceb-8468-6766ded63515-tls-certs\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.902284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.902145 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-model-cache\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.902284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.902222 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-tmp-dir\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.902284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.902275 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfznq\" (UniqueName: \"kubernetes.io/projected/df753c92-98ff-4ceb-8468-6766ded63515-kube-api-access-cfznq\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.902913 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.902352 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-dshm\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.902913 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.902441 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-home\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.902913 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.902609 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-model-cache\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.903079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.902985 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-tmp-dir\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.904617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.904589 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-dshm\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.904871 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.904856 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df753c92-98ff-4ceb-8468-6766ded63515-tls-certs\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.927337 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.927303 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfznq\" (UniqueName: \"kubernetes.io/projected/df753c92-98ff-4ceb-8468-6766ded63515-kube-api-access-cfznq\") pod \"distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:09.949469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:09.949436 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:10.015591 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:10.015513 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" event={"ID":"7bd8ad70-8874-49aa-970f-7f58401c2efe","Type":"ContainerDied","Data":"cc74d7b4ba933a024f1f3881684cf8e7084bb30aed195994a53b93d14cee6c8c"} Oct 30 01:53:10.017858 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:10.015622 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc74d7b4ba933a024f1f3881684cf8e7084bb30aed195994a53b93d14cee6c8c" Oct 30 01:53:10.017858 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:10.015702 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr" Oct 30 01:53:10.269856 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:53:10.269815 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf753c92_98ff_4ceb_8468_6766ded63515.slice/crio-39ac6f7d1adb4a336d1a7b919f158fe79665b30eae6609add10598016577f1b7 WatchSource:0}: Error finding container 39ac6f7d1adb4a336d1a7b919f158fe79665b30eae6609add10598016577f1b7: Status 404 returned error can't find the container with id 39ac6f7d1adb4a336d1a7b919f158fe79665b30eae6609add10598016577f1b7 Oct 30 01:53:10.270579 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:10.270546 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb"] Oct 30 01:53:11.024793 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:11.024759 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" event={"ID":"df753c92-98ff-4ceb-8468-6766ded63515","Type":"ContainerStarted","Data":"73098f44d1f6ff089d382c4d6036ed1fe94f5447c99e1f0ff3cd3b6080816ad3"} Oct 30 01:53:11.024999 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:11.024804 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" event={"ID":"df753c92-98ff-4ceb-8468-6766ded63515","Type":"ContainerStarted","Data":"39ac6f7d1adb4a336d1a7b919f158fe79665b30eae6609add10598016577f1b7"} Oct 30 01:53:11.071745 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:11.071683 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podStartSLOduration=2.071665066 podStartE2EDuration="2.071665066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:11.069518044 +0000 UTC m=+1469.572450282" watchObservedRunningTime="2026-10-30 01:53:11.071665066 +0000 UTC m=+1469.574597287" Oct 30 01:53:11.949644 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:11.949607 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:14.030553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:14.029995 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 30 01:53:14.296753 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:14.296633 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:14.350783 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:14.350729 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:14.402683 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:14.402628 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.019709 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.019640 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.299879 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.299751 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 01:53:15.300916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.300882 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:53:15.361714 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.361659 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.381714 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.381665 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:53:15.393621 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.383291 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.463423 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.463376 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.498411 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.498369 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:53:15.825135 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.825079 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:15.826625 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.826568 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.944552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.941986 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:15.944552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:15.942060 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.022576 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.020670 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.152634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.126300 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.152634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.151222 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:53:16.385554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.356783 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.446670 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.411806 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 01:53:16.446670 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.422657 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.446670 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.427626 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:53:16.509743 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.509672 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:53:16.712675 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.712641 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:53:16.718725 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.718692 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:53:16.753646 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.753606 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded" Oct 30 01:53:16.794566 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.794536 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:53:16.821066 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.821035 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:53:16.823987 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:16.823858 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:53:17.630029 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:53:17.629993 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:53:19.064712 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.064677 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws"] Oct 30 01:53:19.070263 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.070240 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.088758 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.088720 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws"] Oct 30 01:53:19.096608 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.096579 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:19.096771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.096616 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:19.121904 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.121857 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpf7w\" (UniqueName: \"kubernetes.io/projected/dc882188-78c5-4f3b-b982-167506d515cc-kube-api-access-mpf7w\") pod \"maas-api-revoke-sub-f2df25ae-fj8ws\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.122138 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.122115 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2df25ae-fj8ws\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.122249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.122234 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2df25ae-fj8ws\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.224177 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.224134 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2df25ae-fj8ws\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.224437 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.224211 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpf7w\" (UniqueName: \"kubernetes.io/projected/dc882188-78c5-4f3b-b982-167506d515cc-kube-api-access-mpf7w\") pod \"maas-api-revoke-sub-f2df25ae-fj8ws\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.224437 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.224357 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2df25ae-fj8ws\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.225032 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.225005 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2df25ae-fj8ws\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.226862 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.226842 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2df25ae-fj8ws\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.248687 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.248654 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpf7w\" (UniqueName: \"kubernetes.io/projected/dc882188-78c5-4f3b-b982-167506d515cc-kube-api-access-mpf7w\") pod \"maas-api-revoke-sub-f2df25ae-fj8ws\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.386395 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.386305 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:19.701715 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.701679 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws"] Oct 30 01:53:19.702424 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:53:19.702385 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc882188_78c5_4f3b_b982_167506d515cc.slice/crio-92abc1bcb64473350eb1cea17370240a60eb8fbf02b6e661c9bf3059c00c38b5 WatchSource:0}: Error finding container 92abc1bcb64473350eb1cea17370240a60eb8fbf02b6e661c9bf3059c00c38b5: Status 404 returned error can't find the container with id 92abc1bcb64473350eb1cea17370240a60eb8fbf02b6e661c9bf3059c00c38b5 Oct 30 01:53:19.950313 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.950280 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:19.964178 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:19.964135 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:20.419966 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:20.419858 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" event={"ID":"dc882188-78c5-4f3b-b982-167506d515cc","Type":"ContainerStarted","Data":"1fd42eb5dbe625f99e2d2f911ae1014280e9b120fa5f474f7fb98e6e255ad684"} Oct 30 01:53:20.419966 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:20.419931 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" event={"ID":"dc882188-78c5-4f3b-b982-167506d515cc","Type":"ContainerStarted","Data":"92abc1bcb64473350eb1cea17370240a60eb8fbf02b6e661c9bf3059c00c38b5"} Oct 30 01:53:20.431841 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:20.431803 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:20.465799 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:20.465730 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" podStartSLOduration=1.465712511 podStartE2EDuration="1.465712511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:20.462558302 +0000 UTC m=+1478.965490524" watchObservedRunningTime="2026-10-30 01:53:20.465712511 +0000 UTC m=+1478.968644798" Oct 30 01:53:21.429354 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:21.429309 5035 generic.go:356] "Generic (PLEG): container finished" podID="dc882188-78c5-4f3b-b982-167506d515cc" containerID="1fd42eb5dbe625f99e2d2f911ae1014280e9b120fa5f474f7fb98e6e255ad684" exitCode=0 Oct 30 01:53:21.429816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:21.429382 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" event={"ID":"dc882188-78c5-4f3b-b982-167506d515cc","Type":"ContainerDied","Data":"1fd42eb5dbe625f99e2d2f911ae1014280e9b120fa5f474f7fb98e6e255ad684"} Oct 30 01:53:22.610371 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.610340 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:22.670484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.670452 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc882188-78c5-4f3b-b982-167506d515cc-kube-api-access-mpf7w\" (UniqueName: \"kubernetes.io/projected/dc882188-78c5-4f3b-b982-167506d515cc-kube-api-access-mpf7w\") pod \"dc882188-78c5-4f3b-b982-167506d515cc\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " Oct 30 01:53:22.670484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.670486 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-openshift-ca\") pod \"dc882188-78c5-4f3b-b982-167506d515cc\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " Oct 30 01:53:22.670790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.670664 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-xks-ca\") pod \"dc882188-78c5-4f3b-b982-167506d515cc\" (UID: \"dc882188-78c5-4f3b-b982-167506d515cc\") " Oct 30 01:53:22.670930 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.670905 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-openshift-ca" pod "dc882188-78c5-4f3b-b982-167506d515cc" (UID: "dc882188-78c5-4f3b-b982-167506d515cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:22.671344 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.671317 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:22.672941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.672908 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc882188-78c5-4f3b-b982-167506d515cc-kube-api-access-mpf7w" pod "dc882188-78c5-4f3b-b982-167506d515cc" (UID: "dc882188-78c5-4f3b-b982-167506d515cc"). InnerVolumeSpecName "kube-api-access-mpf7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:22.673958 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.673923 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-xks-ca" pod "dc882188-78c5-4f3b-b982-167506d515cc" (UID: "dc882188-78c5-4f3b-b982-167506d515cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:22.772452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.772413 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpf7w\" (UniqueName: \"kubernetes.io/projected/dc882188-78c5-4f3b-b982-167506d515cc-kube-api-access-mpf7w\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:22.772452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:22.772446 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc882188-78c5-4f3b-b982-167506d515cc-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:23.445240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:23.445210 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" Oct 30 01:53:23.445240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:23.445233 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws" event={"ID":"dc882188-78c5-4f3b-b982-167506d515cc","Type":"ContainerDied","Data":"92abc1bcb64473350eb1cea17370240a60eb8fbf02b6e661c9bf3059c00c38b5"} Oct 30 01:53:23.445469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:23.445279 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92abc1bcb64473350eb1cea17370240a60eb8fbf02b6e661c9bf3059c00c38b5" Oct 30 01:53:30.714718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:30.714648 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.719681 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:30.719621 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.797589 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:30.797506 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.876710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:30.876659 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.908689 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:30.908643 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:30.908998 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:30.908966 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.963663 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:30.963610 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.098775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:31.098645 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.393759 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:31.393638 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.463556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:31.454640 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.725564 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:31.717399 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.727858 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:31.726202 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:53:31.831554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:31.815689 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.901965 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:31.899689 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:32.091622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.090409 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:53:32.091622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.090468 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m"] Oct 30 01:53:32.091622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.091393 5035 cpu_manager.go:414] "containerMap: removing container" podUID="dc882188-78c5-4f3b-b982-167506d515cc" containerName="revoke-keys" Oct 30 01:53:32.091622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.091415 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc882188-78c5-4f3b-b982-167506d515cc" containerName="revoke-keys" Oct 30 01:53:32.091987 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.091627 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc882188-78c5-4f3b-b982-167506d515cc" containerName="revoke-keys" Oct 30 01:53:32.110641 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.100741 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m"] Oct 30 01:53:32.110641 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.100884 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:53:32.110641 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.100924 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:53:32.110641 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.100956 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:53:32.110641 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.101824 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:32.111205 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.111161 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:53:32.134407 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.134366 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:32.134940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.134911 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:32.137425 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.137397 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:53:32.461596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.461520 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:32.467569 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.465742 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:32.484747 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.484653 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:53:32.542262 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.542225 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:53:32.624226 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.624198 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:32.717848 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.717736 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:53:32.759274 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.759243 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:53:32.921704 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.921648 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:32.966259 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:32.951634 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b05c63-9zc6m\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.097598 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.097475 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlznc\" (UniqueName: \"kubernetes.io/projected/61d6ac09-9d47-4086-85a5-56a1fa7c101d-kube-api-access-qlznc\") pod \"maas-api-revoke-sub-85b05c63-9zc6m\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.097783 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.097708 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b05c63-9zc6m\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.144557 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.137165 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:33.150727 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.150663 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:53:33.194662 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.192684 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:53:33.200519 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.200476 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlznc\" (UniqueName: \"kubernetes.io/projected/61d6ac09-9d47-4086-85a5-56a1fa7c101d-kube-api-access-qlznc\") pod \"maas-api-revoke-sub-85b05c63-9zc6m\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.200736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.200673 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b05c63-9zc6m\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.200965 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.200944 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b05c63-9zc6m\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.202105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.201981 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b05c63-9zc6m\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.205679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.205616 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b05c63-9zc6m\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.234686 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.234643 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlznc\" (UniqueName: \"kubernetes.io/projected/61d6ac09-9d47-4086-85a5-56a1fa7c101d-kube-api-access-qlznc\") pod \"maas-api-revoke-sub-85b05c63-9zc6m\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.476867 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.476830 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:33.791281 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:33.791239 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m"] Oct 30 01:53:33.792615 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:53:33.792574 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d6ac09_9d47_4086_85a5_56a1fa7c101d.slice/crio-0eff531d5ad5132d7dbd1d38df04145f2960aa7ba1d9a72cce17d764af69852a WatchSource:0}: Error finding container 0eff531d5ad5132d7dbd1d38df04145f2960aa7ba1d9a72cce17d764af69852a: Status 404 returned error can't find the container with id 0eff531d5ad5132d7dbd1d38df04145f2960aa7ba1d9a72cce17d764af69852a Oct 30 01:53:34.769562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:34.769495 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" event={"ID":"61d6ac09-9d47-4086-85a5-56a1fa7c101d","Type":"ContainerStarted","Data":"7f0fd15f37ce62331c22703670efe4d7342f3db5209ca013c89c05b1102b958c"} Oct 30 01:53:34.769716 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:34.769634 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" event={"ID":"61d6ac09-9d47-4086-85a5-56a1fa7c101d","Type":"ContainerStarted","Data":"0eff531d5ad5132d7dbd1d38df04145f2960aa7ba1d9a72cce17d764af69852a"} Oct 30 01:53:34.815417 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:34.815349 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" podStartSLOduration=3.815333475 podStartE2EDuration="3.815333475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:34.812611402 +0000 UTC m=+1493.315543643" watchObservedRunningTime="2026-10-30 01:53:34.815333475 +0000 UTC m=+1493.318265695" Oct 30 01:53:35.778505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:35.778462 5035 generic.go:356] "Generic (PLEG): container finished" podID="61d6ac09-9d47-4086-85a5-56a1fa7c101d" containerID="7f0fd15f37ce62331c22703670efe4d7342f3db5209ca013c89c05b1102b958c" exitCode=0 Oct 30 01:53:35.778710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:35.778583 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" event={"ID":"61d6ac09-9d47-4086-85a5-56a1fa7c101d","Type":"ContainerDied","Data":"7f0fd15f37ce62331c22703670efe4d7342f3db5209ca013c89c05b1102b958c"} Oct 30 01:53:36.186809 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.186699 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km"] Oct 30 01:53:36.193909 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.193875 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.212981 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.212949 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km"] Oct 30 01:53:36.246854 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.246814 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.247076 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.246865 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.247076 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.247036 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.247200 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.247139 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.247200 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.247194 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5fv\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-kube-api-access-dq5fv\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.247310 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.247216 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.247310 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.247267 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.247385 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.247372 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.247417 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.247397 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.348657 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.348616 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.348906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.348690 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.348906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.348711 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5fv\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-kube-api-access-dq5fv\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.348906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.348732 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.348906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.348886 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.349116 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.349018 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.349116 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.349029 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.349116 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.349056 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.349254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.349235 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.349335 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.349233 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.349335 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.349296 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.349435 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.349326 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.349435 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.349417 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.349517 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.349500 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.351412 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.351384 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.351838 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.351814 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.373435 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.373398 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.373651 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.373625 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5fv\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-kube-api-access-dq5fv\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.514265 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.514218 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:36.839014 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:53:36.838974 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4727e030_5504_4bc5_85f0_c9d26fc12aa9.slice/crio-7db3fbf33e18cd36520e5e8744307384ec738fa299d72dae38d158a67b0af49a WatchSource:0}: Error finding container 7db3fbf33e18cd36520e5e8744307384ec738fa299d72dae38d158a67b0af49a: Status 404 returned error can't find the container with id 7db3fbf33e18cd36520e5e8744307384ec738fa299d72dae38d158a67b0af49a Oct 30 01:53:36.842126 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.842075 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:36.842241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.842170 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:36.842241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.842215 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:36.845030 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.845002 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km"] Oct 30 01:53:36.961941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:36.961907 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:37.068487 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.068370 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d6ac09-9d47-4086-85a5-56a1fa7c101d-kube-api-access-qlznc\" (UniqueName: \"kubernetes.io/projected/61d6ac09-9d47-4086-85a5-56a1fa7c101d-kube-api-access-qlznc\") pod \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " Oct 30 01:53:37.068487 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.068485 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-openshift-ca\") pod \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " Oct 30 01:53:37.068766 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.068659 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-xks-ca\") pod \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\" (UID: \"61d6ac09-9d47-4086-85a5-56a1fa7c101d\") " Oct 30 01:53:37.068930 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.068897 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-openshift-ca" pod "61d6ac09-9d47-4086-85a5-56a1fa7c101d" (UID: "61d6ac09-9d47-4086-85a5-56a1fa7c101d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:37.069181 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.069157 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:37.070947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.070919 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-xks-ca" pod "61d6ac09-9d47-4086-85a5-56a1fa7c101d" (UID: "61d6ac09-9d47-4086-85a5-56a1fa7c101d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:37.071055 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.070949 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d6ac09-9d47-4086-85a5-56a1fa7c101d-kube-api-access-qlznc" pod "61d6ac09-9d47-4086-85a5-56a1fa7c101d" (UID: "61d6ac09-9d47-4086-85a5-56a1fa7c101d"). InnerVolumeSpecName "kube-api-access-qlznc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:37.170853 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.170818 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlznc\" (UniqueName: \"kubernetes.io/projected/61d6ac09-9d47-4086-85a5-56a1fa7c101d-kube-api-access-qlznc\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:37.170853 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.170851 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d6ac09-9d47-4086-85a5-56a1fa7c101d-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:37.799984 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.799945 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" event={"ID":"4727e030-5504-4bc5-85f0-c9d26fc12aa9","Type":"ContainerStarted","Data":"873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5"} Oct 30 01:53:37.800471 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.800011 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" event={"ID":"4727e030-5504-4bc5-85f0-c9d26fc12aa9","Type":"ContainerStarted","Data":"7db3fbf33e18cd36520e5e8744307384ec738fa299d72dae38d158a67b0af49a"} Oct 30 01:53:37.801522 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.801493 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" event={"ID":"61d6ac09-9d47-4086-85a5-56a1fa7c101d","Type":"ContainerDied","Data":"0eff531d5ad5132d7dbd1d38df04145f2960aa7ba1d9a72cce17d764af69852a"} Oct 30 01:53:37.801522 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.801501 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m" Oct 30 01:53:37.801708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.801567 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eff531d5ad5132d7dbd1d38df04145f2960aa7ba1d9a72cce17d764af69852a" Oct 30 01:53:37.875356 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:37.875291 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podStartSLOduration=1.875268151 podStartE2EDuration="1.875268151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:37.861975368 +0000 UTC m=+1496.364907588" watchObservedRunningTime="2026-10-30 01:53:37.875268151 +0000 UTC m=+1496.378200372" Oct 30 01:53:38.515193 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:38.515152 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:38.516897 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:38.516867 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:38.517044 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:38.516917 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:39.515655 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:39.515614 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:39.516187 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:39.515674 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:40.122118 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.122083 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx"] Oct 30 01:53:40.122742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.122727 5035 cpu_manager.go:414] "containerMap: removing container" podUID="61d6ac09-9d47-4086-85a5-56a1fa7c101d" containerName="revoke-keys" Oct 30 01:53:40.122809 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.122743 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d6ac09-9d47-4086-85a5-56a1fa7c101d" containerName="revoke-keys" Oct 30 01:53:40.122878 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.122869 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d6ac09-9d47-4086-85a5-56a1fa7c101d" containerName="revoke-keys" Oct 30 01:53:40.126767 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.126748 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.164417 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.164377 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx"] Oct 30 01:53:40.243651 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.243609 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.243860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.243660 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.243860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.243723 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.243860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.243772 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j74g\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-kube-api-access-5j74g\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.243860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.243824 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.244006 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.243898 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.244006 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.243963 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.244070 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.244054 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.244104 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.244077 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.346134 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.346086 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.346460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.346427 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.346639 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.346479 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.346639 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.346513 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j74g\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-kube-api-access-5j74g\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.346639 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.346602 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.346903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.346679 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.346903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.346775 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.346903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.346831 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.346903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.346865 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.348708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.348646 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.349665 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.349191 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.349665 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.349435 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.351827 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.351046 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.353477 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.353447 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.354788 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.354756 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.357573 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.356090 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.399615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.396721 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.412300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.412267 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j74g\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-kube-api-access-5j74g\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.443202 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.443163 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:40.516352 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.515432 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:40.516352 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.515506 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:40.805734 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.805699 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx"] Oct 30 01:53:40.806430 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:53:40.806402 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9e1898_ff05_4c0c_bf18_858d74c2b636.slice/crio-622c350f9a16a665dbece40e6aca568cd3e94f43dac246f6c17e0763e5e3d65f WatchSource:0}: Error finding container 622c350f9a16a665dbece40e6aca568cd3e94f43dac246f6c17e0763e5e3d65f: Status 404 returned error can't find the container with id 622c350f9a16a665dbece40e6aca568cd3e94f43dac246f6c17e0763e5e3d65f Oct 30 01:53:40.809128 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.809095 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:40.809215 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.809172 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:40.809215 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.809205 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:40.849964 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.849928 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" event={"ID":"5c9e1898-ff05-4c0c-bf18-858d74c2b636","Type":"ContainerStarted","Data":"622c350f9a16a665dbece40e6aca568cd3e94f43dac246f6c17e0763e5e3d65f"} Oct 30 01:53:40.955585 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.955394 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz"] Oct 30 01:53:40.960798 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:40.960700 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.008880 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.008840 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz"] Oct 30 01:53:41.059944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.059717 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.059944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.059773 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.059944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.059918 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.060244 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.059980 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.060244 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.060032 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.060244 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.060071 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24815994-57e7-4a2a-9a36-0be361ef7ba1-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.060244 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.060161 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx97n\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-kube-api-access-kx97n\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.060412 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.060383 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.060455 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.060435 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.161736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.161698 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx97n\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-kube-api-access-kx97n\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.161945 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.161811 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.161945 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.161847 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.161945 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.161920 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.162125 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.161962 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.162125 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.162024 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.162125 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.162073 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.162296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.162129 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.162296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.162176 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24815994-57e7-4a2a-9a36-0be361ef7ba1-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.162467 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.162410 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.162542 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.162490 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.162601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.162555 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.163026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.163003 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.163380 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.163354 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24815994-57e7-4a2a-9a36-0be361ef7ba1-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.164736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.164707 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.164905 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.164890 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.196048 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.194182 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.196048 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.195978 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx97n\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-kube-api-access-kx97n\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.294215 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.293945 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:41.518066 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.517942 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:41.518066 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.518018 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:41.627703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.627496 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz"] Oct 30 01:53:41.633781 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.633739 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:41.633874 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.633836 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:41.633916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:41.633876 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:42.268555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.256650 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" event={"ID":"24815994-57e7-4a2a-9a36-0be361ef7ba1","Type":"ContainerStarted","Data":"1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7"} Oct 30 01:53:42.268555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.256744 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" event={"ID":"24815994-57e7-4a2a-9a36-0be361ef7ba1","Type":"ContainerStarted","Data":"04d1d4263786719a96e89eb289492292c60ed0fec0dbe37b4e583b188442ee87"} Oct 30 01:53:42.272556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.270828 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" event={"ID":"5c9e1898-ff05-4c0c-bf18-858d74c2b636","Type":"ContainerStarted","Data":"73c1dd092aa52f0b9a477b1740d14614e55725bb78a14f355098109c57ad5ff3"} Oct 30 01:53:42.302593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.301616 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:42.327555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.315494 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:42.327555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.315587 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:42.410550 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.398500 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podStartSLOduration=2.398475881 podStartE2EDuration="2.398475881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:42.326243953 +0000 UTC m=+1500.829176174" watchObservedRunningTime="2026-10-30 01:53:42.398475881 +0000 UTC m=+1500.901408101" Oct 30 01:53:42.463553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.462731 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:42.473556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.470670 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:42.473556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.470735 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:42.473556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.471686 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podStartSLOduration=2.471672618 podStartE2EDuration="2.471672618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:42.376413186 +0000 UTC m=+1500.879345419" watchObservedRunningTime="2026-10-30 01:53:42.471672618 +0000 UTC m=+1500.974604840" Oct 30 01:53:42.492554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.485202 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 01:53:42.551617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.550979 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:42.551617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.551045 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:42.589550 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:42.582916 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 01:53:43.144887 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.144832 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:53:43.365556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.328284 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:53:43.365556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.332069 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:43.365556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.332133 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:43.365556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.335139 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:53:43.365556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.354184 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.400661 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.377619 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.444971 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.444931 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:43.446221 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.446179 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:43.531555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.518646 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.531555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.522376 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:43.531555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.522523 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:43.608559 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.608017 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.685617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.672773 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.733691 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.733634 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.930700 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:43.930639 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.057254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.056908 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.149277 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.146661 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.149277 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.147601 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:53:44.551628 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.480728 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.551628 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.481220 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.631145 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.631231 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.631996 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.632316 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.639120 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.639867 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.659188 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.659262 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.659343 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.659368 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.703668 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.727220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.704299 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:44.904622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.880084 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.031555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:44.946332 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.092593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.039672 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:53:45.237617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.212137 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:53:45.238345 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.159689 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.305934 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.305890 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:45.306186 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.306153 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:45.495685 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.494713 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:45.525754 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.525709 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:45.527466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.527429 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 01:53:45.556111 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.556057 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:45.556419 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.556369 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:45.652462 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.652395 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.654273 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.654218 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.713153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.704728 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 01:53:45.957320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.957260 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:45.958053 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.957923 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.958275 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.957998 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:45.958452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:45.958431 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.079920 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.079857 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:53:46.094360 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.094312 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.317615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.305961 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:46.317615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.306028 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:46.485557 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.442746 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:53:46.485557 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.445106 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:46.485557 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.445162 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:46.612555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.612423 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.615306 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.615268 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:46.635564 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.628563 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:46.654949 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.654902 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:53:46.660675 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.660632 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.692620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.692576 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.704764 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.704700 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.705992 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:46.705951 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.098017 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.097687 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.212099 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.211972 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.437112 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.302232 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:47.437386 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.437352 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:47.555647 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.549265 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:53:47.555647 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.550359 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:47.555647 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.550420 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:47.555647 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.550975 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:53:47.555647 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.551033 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:53:47.613234 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.613169 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.746779 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.735088 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.763569 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.748253 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:48.246565 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:47.856147 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:48.250013 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.131266 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 01:53:48.252701 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.252661 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:48.362922 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:53:48.362697 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:53:48.443736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.437715 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:53:48.443736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.438166 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:53:48.443736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.438239 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:53:48.443736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.438288 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:53:48.447423 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.447381 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:48.447636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.447447 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:48.449363 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.449331 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp"] Oct 30 01:53:48.452363 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.452328 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:48.452511 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.452386 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:48.465824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.464792 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:53:48.465824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.464852 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:53:48.465824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.464873 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp"] Oct 30 01:53:48.465824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.464900 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.465824 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.465733 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:53:48.503971 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.503860 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:48.535496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.534562 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:48.645975 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.645939 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a2dfaee-gcbdp\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.646441 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.646422 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tdhd\" (UniqueName: \"kubernetes.io/projected/fd698533-6fa7-4af0-9775-ea05d2a896e4-kube-api-access-4tdhd\") pod \"maas-api-revoke-sub-6a2dfaee-gcbdp\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.646671 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.646642 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a2dfaee-gcbdp\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.748452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.748417 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tdhd\" (UniqueName: \"kubernetes.io/projected/fd698533-6fa7-4af0-9775-ea05d2a896e4-kube-api-access-4tdhd\") pod \"maas-api-revoke-sub-6a2dfaee-gcbdp\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.748906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.748859 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a2dfaee-gcbdp\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.750141 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.749725 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a2dfaee-gcbdp\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.750690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.750660 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a2dfaee-gcbdp\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.756092 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.754013 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a2dfaee-gcbdp\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.816554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.797383 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tdhd\" (UniqueName: \"kubernetes.io/projected/fd698533-6fa7-4af0-9775-ea05d2a896e4-kube-api-access-4tdhd\") pod \"maas-api-revoke-sub-6a2dfaee-gcbdp\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:48.846936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:48.846902 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:49.296626 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:49.296109 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:49.296626 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:49.296177 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:49.445670 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:49.445557 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:49.445670 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:49.445633 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:49.592566 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:49.590670 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:49.592566 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:49.590746 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:49.745581 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:49.742373 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp"] Oct 30 01:53:49.747300 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:53:49.746016 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd698533_6fa7_4af0_9775_ea05d2a896e4.slice/crio-2991bae93778d7ba408735c44b482e14e1882ac4f47a1fb9e5ed21fb93bff4c3 WatchSource:0}: Error finding container 2991bae93778d7ba408735c44b482e14e1882ac4f47a1fb9e5ed21fb93bff4c3: Status 404 returned error can't find the container with id 2991bae93778d7ba408735c44b482e14e1882ac4f47a1fb9e5ed21fb93bff4c3 Oct 30 01:53:50.300554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.295579 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:50.300554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.295660 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:50.354355 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.354268 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:50.354766 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.354740 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:50.355522 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.355501 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:53:50.443896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.443833 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:50.443896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.443890 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:50.492590 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.466847 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" event={"ID":"fd698533-6fa7-4af0-9775-ea05d2a896e4","Type":"ContainerStarted","Data":"c77b5d0d286d5bdc6a79a82aa3d8566224b3a2a13bba2951e3e9beb633041a13"} Oct 30 01:53:50.492590 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.466960 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" event={"ID":"fd698533-6fa7-4af0-9775-ea05d2a896e4","Type":"ContainerStarted","Data":"2991bae93778d7ba408735c44b482e14e1882ac4f47a1fb9e5ed21fb93bff4c3"} Oct 30 01:53:50.558175 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:50.558095 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" podStartSLOduration=2.558068645 podStartE2EDuration="2.558068645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:50.541485766 +0000 UTC m=+1509.044418003" watchObservedRunningTime="2026-10-30 01:53:50.558068645 +0000 UTC m=+1509.061000866" Oct 30 01:53:51.295150 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:51.295051 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:51.295424 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:51.295394 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:51.443638 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:51.443591 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:53:51.444140 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:51.443656 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 01:53:51.499642 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:51.499596 5035 generic.go:356] "Generic (PLEG): container finished" podID="fd698533-6fa7-4af0-9775-ea05d2a896e4" containerID="c77b5d0d286d5bdc6a79a82aa3d8566224b3a2a13bba2951e3e9beb633041a13" exitCode=0 Oct 30 01:53:51.499829 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:51.499803 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" event={"ID":"fd698533-6fa7-4af0-9775-ea05d2a896e4","Type":"ContainerDied","Data":"c77b5d0d286d5bdc6a79a82aa3d8566224b3a2a13bba2951e3e9beb633041a13"} Oct 30 01:53:52.295254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.295211 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:53:52.295484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.295297 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:53:52.711615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.711584 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:52.717639 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.717602 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-xks-ca\") pod \"fd698533-6fa7-4af0-9775-ea05d2a896e4\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " Oct 30 01:53:52.718548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.717823 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-openshift-ca\") pod \"fd698533-6fa7-4af0-9775-ea05d2a896e4\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " Oct 30 01:53:52.718548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.717869 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd698533-6fa7-4af0-9775-ea05d2a896e4-kube-api-access-4tdhd\" (UniqueName: \"kubernetes.io/projected/fd698533-6fa7-4af0-9775-ea05d2a896e4-kube-api-access-4tdhd\") pod \"fd698533-6fa7-4af0-9775-ea05d2a896e4\" (UID: \"fd698533-6fa7-4af0-9775-ea05d2a896e4\") " Oct 30 01:53:52.718548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.718241 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-openshift-ca" pod "fd698533-6fa7-4af0-9775-ea05d2a896e4" (UID: "fd698533-6fa7-4af0-9775-ea05d2a896e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:52.718883 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.718726 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:52.719876 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.719847 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-xks-ca" pod "fd698533-6fa7-4af0-9775-ea05d2a896e4" (UID: "fd698533-6fa7-4af0-9775-ea05d2a896e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:52.720489 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.720456 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd698533-6fa7-4af0-9775-ea05d2a896e4-kube-api-access-4tdhd" pod "fd698533-6fa7-4af0-9775-ea05d2a896e4" (UID: "fd698533-6fa7-4af0-9775-ea05d2a896e4"). InnerVolumeSpecName "kube-api-access-4tdhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:52.820637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.820593 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tdhd\" (UniqueName: \"kubernetes.io/projected/fd698533-6fa7-4af0-9775-ea05d2a896e4-kube-api-access-4tdhd\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:52.820637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:52.820632 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd698533-6fa7-4af0-9775-ea05d2a896e4-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:53:53.299297 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.299248 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:53.299729 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.299709 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:53.300653 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.300628 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 01:53:53.485091 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.484689 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:53.485091 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.484764 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:53.526935 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.526899 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" Oct 30 01:53:53.526935 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.526915 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp" event={"ID":"fd698533-6fa7-4af0-9775-ea05d2a896e4","Type":"ContainerDied","Data":"2991bae93778d7ba408735c44b482e14e1882ac4f47a1fb9e5ed21fb93bff4c3"} Oct 30 01:53:53.527226 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.526995 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2991bae93778d7ba408735c44b482e14e1882ac4f47a1fb9e5ed21fb93bff4c3" Oct 30 01:53:53.704450 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.704411 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:53.704873 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.704848 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:53.705928 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:53.705900 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 01:53:57.540331 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.540285 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4"] Oct 30 01:53:57.541345 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.541322 5035 cpu_manager.go:414] "containerMap: removing container" podUID="fd698533-6fa7-4af0-9775-ea05d2a896e4" containerName="revoke-keys" Oct 30 01:53:57.541406 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.541349 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd698533-6fa7-4af0-9775-ea05d2a896e4" containerName="revoke-keys" Oct 30 01:53:57.541545 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.541519 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd698533-6fa7-4af0-9775-ea05d2a896e4" containerName="revoke-keys" Oct 30 01:53:57.545469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.545450 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.576357 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.576317 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"distinct-2-w4-b93d51-kserve-self-signed-certs\"" Oct 30 01:53:57.581569 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.581502 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4"] Oct 30 01:53:57.591216 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.591178 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-model-cache\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.591369 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.591260 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-home\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.591369 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.591292 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tmp-dir\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.591369 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.591348 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n46xq\" (UniqueName: \"kubernetes.io/projected/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-kube-api-access-n46xq\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.591516 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.591487 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-dshm\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.591594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.591514 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tls-certs\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.693232 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.693194 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-model-cache\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.693439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.693253 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-home\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.693439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.693270 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tmp-dir\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.693439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.693303 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n46xq\" (UniqueName: \"kubernetes.io/projected/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-kube-api-access-n46xq\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.693439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.693416 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-dshm\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.693689 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.693443 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tls-certs\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.693750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.693694 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-model-cache\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.693797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.693769 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-home\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.693835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.693798 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tmp-dir\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.695882 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.695852 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-dshm\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.696151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.696133 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tls-certs\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.719325 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.719287 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n46xq\" (UniqueName: \"kubernetes.io/projected/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-kube-api-access-n46xq\") pod \"distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:57.866606 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:57.866455 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:53:58.183917 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:58.183874 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4"] Oct 30 01:53:58.185164 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:53:58.185133 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a1a7bde_671c_4b7f_96d9_a79e994f56c3.slice/crio-9057d6ee189374b5c4a0b9dbba96bdd92e71430d1d25d7f39f43614719e296ff WatchSource:0}: Error finding container 9057d6ee189374b5c4a0b9dbba96bdd92e71430d1d25d7f39f43614719e296ff: Status 404 returned error can't find the container with id 9057d6ee189374b5c4a0b9dbba96bdd92e71430d1d25d7f39f43614719e296ff Oct 30 01:53:58.575296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:58.575262 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" event={"ID":"1a1a7bde-671c-4b7f-96d9-a79e994f56c3","Type":"ContainerStarted","Data":"2ea917afb0365425834fbdf7b64c52bc2cd22cf93e841e8936dbb36aaabad861"} Oct 30 01:53:58.575760 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:58.575306 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" event={"ID":"1a1a7bde-671c-4b7f-96d9-a79e994f56c3","Type":"ContainerStarted","Data":"9057d6ee189374b5c4a0b9dbba96bdd92e71430d1d25d7f39f43614719e296ff"} Oct 30 01:53:58.633650 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:58.633563 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" podStartSLOduration=1.6335395799999999 podStartE2EDuration="1.63353958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:58.632056492 +0000 UTC m=+1517.134988733" watchObservedRunningTime="2026-10-30 01:53:58.63353958 +0000 UTC m=+1517.136471802" Oct 30 01:53:58.867336 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:53:58.867252 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:54:02.451555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:02.451011 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:02.474555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:02.459858 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 01:54:02.637554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:02.625695 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:02.683555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:02.682718 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:02.683555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:02.683130 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 01:54:03.098581 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.096335 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.102555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.101624 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.349615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.348776 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.456555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.455872 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.457136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.456907 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:54:03.464554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.460105 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 01:54:03.466258 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.466222 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:54:03.709624 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.700040 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 01:54:03.773555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.769678 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.773555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.770099 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:54:03.860722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.860350 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:03.860722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:03.860425 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.067624 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.052298 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 30 01:54:04.110623 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.097114 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.111351 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.111302 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 01:54:04.297079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.297037 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:54:04.297719 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.297681 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:54:04.328643 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.327336 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:04.328643 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.327411 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.340701 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.340664 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:54:04.383567 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.381044 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:04.383567 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.381118 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.416209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.416179 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:54:04.454332 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.454288 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:54:04.480592 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.477679 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 01:54:04.481787 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.481453 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.755554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.754611 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.782549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.782449 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.826886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:04.826814 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.131888 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:05.131745 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:54:05.132640 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:05.132593 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 01:54:05.493555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:05.492637 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.532557 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:05.530676 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.774866 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:05.774753 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 01:54:05.804368 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:05.804320 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.861917 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:05.861874 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:54:06.169620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.144338 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:06.171022 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.160616 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 01:54:06.474884 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.474837 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:06.475215 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.475179 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:06.497225 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.497176 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:06.580563 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.552727 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:54:06.749093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.706700 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 30 01:54:06.749410 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.725820 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded" Oct 30 01:54:06.812173 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.803103 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:06.812173 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.806476 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 01:54:06.872205 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.872156 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:06.886562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:06.885929 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v"] Oct 30 01:54:07.215555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.202621 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 01:54:07.247556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.216355 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:54:07.267551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.228889 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:07.310596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.310254 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v\" (UID: \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:07.310596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.310367 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4m4\" (UniqueName: \"kubernetes.io/projected/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-kube-api-access-dm4m4\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v\" (UID: \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:07.322895 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.322737 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2933ff00\"" Oct 30 01:54:07.322895 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.322857 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2933ff00-dockercfg-tgxf9\"" Oct 30 01:54:07.461519 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.415098 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v\" (UID: \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:07.469340 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.464085 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4m4\" (UniqueName: \"kubernetes.io/projected/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-kube-api-access-dm4m4\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v\" (UID: \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:07.474737 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.423795 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v\" (UID: \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:07.479549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.468044 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v"] Oct 30 01:54:07.554586 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.553881 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:07.588825 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.588766 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:54:07.799616 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.686382 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4m4\" (UniqueName: \"kubernetes.io/projected/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-kube-api-access-dm4m4\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v\" (UID: \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:07.799616 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.740303 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:07.804554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.803342 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:54:07.804554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.803429 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:54:07.804554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.803498 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:07.852480 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.852412 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:07.936698 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.936659 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:54:07.985166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:07.985044 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:54:08.286553 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:08.260895 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:08.565612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:08.564678 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:54:08.606595 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:08.606035 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:08.616973 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:08.616934 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:54:08.618619 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:08.618584 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:08.703707 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:08.702975 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:54:08.831614 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:08.830617 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:09.018439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.018371 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:09.084896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.084171 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:54:09.232809 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.232774 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:54:09.238179 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.238074 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:54:09.239604 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.239176 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:54:09.616290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.616254 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v"] Oct 30 01:54:09.619144 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:54:09.619107 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4dd90b_5e24_44aa_a3a3_0369d58f65e8.slice/crio-27a45658a156e41d8ec8f48faba0521f7877e85544335ab28bef5da41f53fe3e WatchSource:0}: Error finding container 27a45658a156e41d8ec8f48faba0521f7877e85544335ab28bef5da41f53fe3e: Status 404 returned error can't find the container with id 27a45658a156e41d8ec8f48faba0521f7877e85544335ab28bef5da41f53fe3e Oct 30 01:54:09.839636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.839601 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc"] Oct 30 01:54:09.844119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.844095 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:09.863249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.863214 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2198e4\"" Oct 30 01:54:09.863435 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.863417 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2198e4-dockercfg-87jgp\"" Oct 30 01:54:09.865506 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.865439 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc"] Oct 30 01:54:09.980659 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.980621 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7"] Oct 30 01:54:09.991946 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:09.991915 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:10.020575 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.010098 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7"] Oct 30 01:54:10.021574 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.018939 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d3b9-abfd-4371-9c89-d1badb083c78-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc\" (UID: \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:10.021867 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.021841 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spc78\" (UniqueName: \"kubernetes.io/projected/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-kube-api-access-spc78\") pod \"payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7\" (UID: \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:10.022120 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.022100 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7\" (UID: \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:10.022290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.022274 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmpxb\" (UniqueName: \"kubernetes.io/projected/cdd4d3b9-abfd-4371-9c89-d1badb083c78-kube-api-access-dmpxb\") pod \"payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc\" (UID: \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:10.125309 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.124253 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpxb\" (UniqueName: \"kubernetes.io/projected/cdd4d3b9-abfd-4371-9c89-d1badb083c78-kube-api-access-dmpxb\") pod \"payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc\" (UID: \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:10.125309 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.124417 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d3b9-abfd-4371-9c89-d1badb083c78-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc\" (UID: \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:10.125309 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.124477 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spc78\" (UniqueName: \"kubernetes.io/projected/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-kube-api-access-spc78\") pod \"payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7\" (UID: \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:10.125309 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.124577 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7\" (UID: \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:10.125684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.125418 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7\" (UID: \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:10.125684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.125441 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d3b9-abfd-4371-9c89-d1badb083c78-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc\" (UID: \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:10.149387 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.149352 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spc78\" (UniqueName: \"kubernetes.io/projected/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-kube-api-access-spc78\") pod \"payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7\" (UID: \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:10.150038 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.150009 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmpxb\" (UniqueName: \"kubernetes.io/projected/cdd4d3b9-abfd-4371-9c89-d1badb083c78-kube-api-access-dmpxb\") pod \"payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc\" (UID: \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:10.158850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.158813 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:10.204906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.204859 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" event={"ID":"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8","Type":"ContainerStarted","Data":"b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088"} Oct 30 01:54:10.205110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.204977 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:10.205110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.205003 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" event={"ID":"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8","Type":"ContainerStarted","Data":"27a45658a156e41d8ec8f48faba0521f7877e85544335ab28bef5da41f53fe3e"} Oct 30 01:54:10.262189 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.254799 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" podStartSLOduration=4.254782984 podStartE2EDuration="4.254782984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:10.248881802 +0000 UTC m=+1528.751814026" watchObservedRunningTime="2026-10-30 01:54:10.254782984 +0000 UTC m=+1528.757715254" Oct 30 01:54:10.330509 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.330465 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:10.492705 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.492669 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc"] Oct 30 01:54:10.493251 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:54:10.493216 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd4d3b9_abfd_4371_9c89_d1badb083c78.slice/crio-0f19811203ebb0780794c80a744f42a36dab8386daa5a9751c1685209ecb5603 WatchSource:0}: Error finding container 0f19811203ebb0780794c80a744f42a36dab8386daa5a9751c1685209ecb5603: Status 404 returned error can't find the container with id 0f19811203ebb0780794c80a744f42a36dab8386daa5a9751c1685209ecb5603 Oct 30 01:54:10.655822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:10.655789 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7"] Oct 30 01:54:10.656418 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:54:10.656385 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741f058f_d2ee_40d8_9f9d_6fac9c19e3ae.slice/crio-12d05b47e0f0cc314e2ae91b05e74adbadcc8c5216227a0a703262db2b529f84 WatchSource:0}: Error finding container 12d05b47e0f0cc314e2ae91b05e74adbadcc8c5216227a0a703262db2b529f84: Status 404 returned error can't find the container with id 12d05b47e0f0cc314e2ae91b05e74adbadcc8c5216227a0a703262db2b529f84 Oct 30 01:54:11.215228 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:11.215186 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" event={"ID":"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae","Type":"ContainerStarted","Data":"a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce"} Oct 30 01:54:11.215438 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:11.215255 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" event={"ID":"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae","Type":"ContainerStarted","Data":"12d05b47e0f0cc314e2ae91b05e74adbadcc8c5216227a0a703262db2b529f84"} Oct 30 01:54:11.215438 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:11.215284 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:11.217107 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:11.217079 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" event={"ID":"cdd4d3b9-abfd-4371-9c89-d1badb083c78","Type":"ContainerStarted","Data":"89d4cf7bff88e54f80e51d9fa2ae0a55c37e454f08598e18bb6903d1a42bcf9b"} Oct 30 01:54:11.217231 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:11.217128 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" event={"ID":"cdd4d3b9-abfd-4371-9c89-d1badb083c78","Type":"ContainerStarted","Data":"0f19811203ebb0780794c80a744f42a36dab8386daa5a9751c1685209ecb5603"} Oct 30 01:54:11.217298 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:11.217252 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:11.261829 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:11.261765 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" podStartSLOduration=2.261750999 podStartE2EDuration="2.261750999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:11.260121752 +0000 UTC m=+1529.763053975" watchObservedRunningTime="2026-10-30 01:54:11.261750999 +0000 UTC m=+1529.764683268" Oct 30 01:54:11.306360 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:11.306296 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" podStartSLOduration=2.306278422 podStartE2EDuration="2.306278422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:11.302468772 +0000 UTC m=+1529.805400992" watchObservedRunningTime="2026-10-30 01:54:11.306278422 +0000 UTC m=+1529.809210641" Oct 30 01:54:16.702310 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.702270 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb"] Oct 30 01:54:16.708138 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.708103 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.731105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.731072 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2933ff00-serving-cert\"" Oct 30 01:54:16.736081 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.735635 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb"] Oct 30 01:54:16.818989 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.818943 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-metrics-certs\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.818989 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.819000 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-maas-api-tls\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.819246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.819173 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6ffb31-b827-408e-b303-7b3060a17d1f-openshift-service-ca\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.819246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.819216 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn85j\" (UniqueName: \"kubernetes.io/projected/6a6ffb31-b827-408e-b303-7b3060a17d1f-kube-api-access-vn85j\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.921012 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.920912 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-metrics-certs\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.921199 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.921049 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-maas-api-tls\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.921199 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.921187 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6ffb31-b827-408e-b303-7b3060a17d1f-openshift-service-ca\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.921496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.921234 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn85j\" (UniqueName: \"kubernetes.io/projected/6a6ffb31-b827-408e-b303-7b3060a17d1f-kube-api-access-vn85j\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.922062 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.922028 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6ffb31-b827-408e-b303-7b3060a17d1f-openshift-service-ca\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.924839 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.924159 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-metrics-certs\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.924839 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.924672 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-maas-api-tls\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:16.946574 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:16.946521 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn85j\" (UniqueName: \"kubernetes.io/projected/6a6ffb31-b827-408e-b303-7b3060a17d1f-kube-api-access-vn85j\") pod \"maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:17.027747 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:17.027645 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:17.343190 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:17.343152 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb"] Oct 30 01:54:17.343620 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:54:17.343588 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6ffb31_b827_408e_b303_7b3060a17d1f.slice/crio-84ea1696b4f61dea73c29c9937475fe7266cbe42b414ce410ab06ddf4d7595bb WatchSource:0}: Error finding container 84ea1696b4f61dea73c29c9937475fe7266cbe42b414ce410ab06ddf4d7595bb: Status 404 returned error can't find the container with id 84ea1696b4f61dea73c29c9937475fe7266cbe42b414ce410ab06ddf4d7595bb Oct 30 01:54:17.345247 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:17.345226 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:54:17.630143 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:54:17.630041 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:54:18.273815 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:18.273774 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" event={"ID":"6a6ffb31-b827-408e-b303-7b3060a17d1f","Type":"ContainerStarted","Data":"4c006412f4f8e4603fb840b59bdd843b8602a0f1c723701ad05bb7c6be762374"} Oct 30 01:54:18.274342 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:18.273831 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" event={"ID":"6a6ffb31-b827-408e-b303-7b3060a17d1f","Type":"ContainerStarted","Data":"84ea1696b4f61dea73c29c9937475fe7266cbe42b414ce410ab06ddf4d7595bb"} Oct 30 01:54:18.274342 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:18.273906 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:18.323676 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:18.323611 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" podStartSLOduration=1.915848056 podStartE2EDuration="2.323592959s" totalImagesPullingTime="407.744903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:18.320683185 +0000 UTC m=+1536.823615442" watchObservedRunningTime="2026-10-30 01:54:18.323592959 +0000 UTC m=+1536.826525179" Oct 30 01:54:20.141555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:20.135041 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.178059 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:20.176722 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.526555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:20.521839 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.526555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:20.522714 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.788618 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:20.787657 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 01:54:20.839555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:20.838641 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.839555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:20.839102 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:54:20.874879 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:20.874754 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" podUID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 01:54:20.999058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:20.998625 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.178418 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.178294 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:54:21.179565 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.179407 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:54:21.226767 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.226731 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:54:21.525679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.525628 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:54:21.680554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.541266 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 01:54:21.680554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.651843 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:54:21.680554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.651954 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:54:21.813268 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.813158 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.839446 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.839396 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.866804 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.866737 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.923452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:21.918075 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" podUID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:22.203660 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.186855 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:54:22.228066 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.220761 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:22.481001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.470814 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:54:22.481001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.470912 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:54:22.481001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.470971 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:54:22.481001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.471026 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:54:22.651144 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.651087 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:22.676555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.652605 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:54:22.801874 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.801772 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:54:22.982059 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:22.982025 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 01:54:23.196271 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.155088 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:54:23.386456 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.386389 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:23.387431 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.386688 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:54:23.387647 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.386750 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 01:54:23.387797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.386814 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 01:54:23.387934 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.386842 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" podUID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 01:54:23.388066 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.386891 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 01:54:23.443685 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.443657 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:54:23.475436 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.475383 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:23.760605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.742405 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:54:23.986597 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:23.986503 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:24.063885 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:24.060322 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:54:24.064236 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:24.064210 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 01:54:24.064448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:24.064424 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:54:24.064629 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:24.064615 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:54:24.065180 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:24.065162 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:54:24.244892 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:24.244851 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:54:24.356697 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:24.349688 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:54:24.363192 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:24.363151 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:54:36.142466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.142420 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l"] Oct 30 01:54:36.146911 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.146886 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.167516 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.167487 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 01:54:36.167713 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.167588 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 01:54:36.167713 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.167634 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 01:54:36.177552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.177490 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v6t8d\"" Oct 30 01:54:36.210002 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.209963 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l"] Oct 30 01:54:36.261502 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.261461 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.261742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.261626 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.261742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.261731 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.261872 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.261855 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.261925 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.261885 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkc9m\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-kube-api-access-rkc9m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.261983 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.261950 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.262037 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.261993 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.363936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.363879 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.364136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.364028 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.364136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.364099 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.364257 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.364231 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.364311 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.364259 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkc9m\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-kube-api-access-rkc9m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.364311 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.364288 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.364420 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.364327 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.365886 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.365522 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.368021 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.367993 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.368171 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.368147 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.368477 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.368452 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.369163 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.369139 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.402366 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.402284 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.404982 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.404945 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkc9m\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-kube-api-access-rkc9m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr74l\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.461778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.461739 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:36.796818 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:36.796790 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l"] Oct 30 01:54:36.798007 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:54:36.797976 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f6104d_2e09_41f1_8bc8_6e2fe8c28160.slice/crio-e1195e2d45ffbf510ffcf361e29ea77e608574578fe1b2fa39afbed52abde40f WatchSource:0}: Error finding container e1195e2d45ffbf510ffcf361e29ea77e608574578fe1b2fa39afbed52abde40f: Status 404 returned error can't find the container with id e1195e2d45ffbf510ffcf361e29ea77e608574578fe1b2fa39afbed52abde40f Oct 30 01:54:37.650353 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:37.650315 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" event={"ID":"85f6104d-2e09-41f1-8bc8-6e2fe8c28160","Type":"ContainerStarted","Data":"e1195e2d45ffbf510ffcf361e29ea77e608574578fe1b2fa39afbed52abde40f"} Oct 30 01:54:40.255988 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:40.255947 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:54:40.256391 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:40.256016 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:54:40.677775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:40.677670 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" event={"ID":"85f6104d-2e09-41f1-8bc8-6e2fe8c28160","Type":"ContainerStarted","Data":"37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4"} Oct 30 01:54:40.677930 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:40.677784 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:40.757664 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:40.757577 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" podStartSLOduration=1.303063112 podStartE2EDuration="4.757555089s" totalImagesPullingTime="3.454491977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:40.738713528 +0000 UTC m=+1559.241645741" watchObservedRunningTime="2026-10-30 01:54:40.757555089 +0000 UTC m=+1559.260487305" Oct 30 01:54:41.173847 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.173811 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9"] Oct 30 01:54:41.179059 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.179033 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.211344 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.211290 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9"] Oct 30 01:54:41.272792 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.272743 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.273321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.272823 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.273321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.272889 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.273321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.273064 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.273321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.273124 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.273321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.273168 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a38459-65bc-418d-93e9-13f88b7f9c7d-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.273321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.273271 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdsbc\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-kube-api-access-qdsbc\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.273321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.273319 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.273666 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.273335 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.374943 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.374904 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.375168 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.374969 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.375168 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.374995 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.375168 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.375087 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.375168 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.375125 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.375168 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.375165 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a38459-65bc-418d-93e9-13f88b7f9c7d-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.375494 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.375224 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdsbc\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-kube-api-access-qdsbc\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.375494 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.375285 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.375494 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.375308 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.375881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.375854 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.376137 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.376118 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.379034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.377013 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.379034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.377752 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a38459-65bc-418d-93e9-13f88b7f9c7d-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.379034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.377945 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.385129 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.385087 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.387002 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.386949 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.405004 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.404967 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdsbc\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-kube-api-access-qdsbc\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.406913 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.406882 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.498324 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.498277 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:41.687637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.687608 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 01:54:41.828771 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.828734 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9"] Oct 30 01:54:41.828916 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:54:41.828821 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a38459_65bc_418d_93e9_13f88b7f9c7d.slice/crio-d0de6d70d0f1dd9925707461bd40249c9d842baf58d5e5da0b365a650033c2b7 WatchSource:0}: Error finding container d0de6d70d0f1dd9925707461bd40249c9d842baf58d5e5da0b365a650033c2b7: Status 404 returned error can't find the container with id d0de6d70d0f1dd9925707461bd40249c9d842baf58d5e5da0b365a650033c2b7 Oct 30 01:54:41.831686 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.831650 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:54:41.831794 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.831732 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:54:41.831794 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:41.831764 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:54:42.443715 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:42.443682 5035 scope.go:122] "RemoveContainer" containerID="9b78ad0bac01880b622561e3993f651915a21f85431d242562bbfac4e8229a24" Oct 30 01:54:42.698034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:42.697930 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" event={"ID":"41a38459-65bc-418d-93e9-13f88b7f9c7d","Type":"ContainerStarted","Data":"aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d"} Oct 30 01:54:42.698034 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:42.698000 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" event={"ID":"41a38459-65bc-418d-93e9-13f88b7f9c7d","Type":"ContainerStarted","Data":"d0de6d70d0f1dd9925707461bd40249c9d842baf58d5e5da0b365a650033c2b7"} Oct 30 01:54:42.764796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:42.764728 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" podStartSLOduration=1.764705413 podStartE2EDuration="1.764705413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:42.759848237 +0000 UTC m=+1561.262780471" watchObservedRunningTime="2026-10-30 01:54:42.764705413 +0000 UTC m=+1561.267637633" Oct 30 01:54:43.498488 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:43.498447 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:43.500234 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:43.500199 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:54:43.500361 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:43.500256 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:54:44.499695 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:44.499652 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:54:44.500153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:44.499719 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:54:45.498980 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:45.498901 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:54:45.498980 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:45.498959 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:54:46.498923 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:46.498877 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:54:46.499399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:46.498950 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:54:47.499284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:47.499239 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:54:47.499847 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:47.499309 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:54:47.631227 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:54:47.631186 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:54:48.498966 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:48.498928 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:54:48.499193 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:48.498997 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:54:49.554924 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:49.554873 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:49.557424 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:49.555365 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:49.557424 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:49.556285 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 01:54:59.553510 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.553414 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7"] Oct 30 01:54:59.558240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.558213 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.587348 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.587311 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7"] Oct 30 01:54:59.637382 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.637331 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.637632 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.637429 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.637722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.637650 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.637722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.637692 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmff\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-kube-api-access-zvmff\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.637939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.637760 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.637939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.637803 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.637939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.637825 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.637939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.637931 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.638168 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.638022 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.657072 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.657039 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2198e4-openshift-default-dockercfg-zhwrb\"" Oct 30 01:54:59.739202 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739169 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739216 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmff\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-kube-api-access-zvmff\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739254 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739302 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739336 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739389 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739368 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739630 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739414 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739630 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739609 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739717 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739657 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739717 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739691 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739803 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739720 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.739961 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.739938 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.740039 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.740015 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.740376 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.740341 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.741848 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.741824 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.741958 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.741943 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.780509 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.780475 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmff\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-kube-api-access-zvmff\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.780694 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.780478 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:54:59.876680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:54:59.876597 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:55:00.207279 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:00.207242 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7"] Oct 30 01:55:00.208767 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:55:00.208736 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6d4a1f_957e_47d8_bad7_81b9b68af2e7.slice/crio-185f2aace711f1ab952cfaed7cb7fc8d28ea6c40bca90c34e9d3f5998f7e2366 WatchSource:0}: Error finding container 185f2aace711f1ab952cfaed7cb7fc8d28ea6c40bca90c34e9d3f5998f7e2366: Status 404 returned error can't find the container with id 185f2aace711f1ab952cfaed7cb7fc8d28ea6c40bca90c34e9d3f5998f7e2366 Oct 30 01:55:00.211404 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:00.211367 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:00.211543 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:00.211461 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:00.211543 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:00.211508 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:00.899855 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:00.899816 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" event={"ID":"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7","Type":"ContainerStarted","Data":"1c697babf527ef5348155efcf8dee39c95288fa1cbd9f9f1884beedb4c606b4f"} Oct 30 01:55:00.900373 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:00.899873 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" event={"ID":"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7","Type":"ContainerStarted","Data":"185f2aace711f1ab952cfaed7cb7fc8d28ea6c40bca90c34e9d3f5998f7e2366"} Oct 30 01:55:00.995362 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:00.995289 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" podStartSLOduration=1.995267863 podStartE2EDuration="1.995267863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:00.960613098 +0000 UTC m=+1579.463545310" watchObservedRunningTime="2026-10-30 01:55:00.995267863 +0000 UTC m=+1579.498200084" Oct 30 01:55:01.884556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:01.882974 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:55:01.892058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:01.891685 5035 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:55:01.892058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:01.891756 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 01:55:02.889551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:02.888849 5035 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:55:02.889551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:02.888912 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 01:55:03.887552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:03.887467 5035 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:55:03.887790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:03.887585 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 01:55:04.887507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:04.887018 5035 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:55:04.887507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:04.887090 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 01:55:05.891750 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:05.891709 5035 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:55:05.892312 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:05.892138 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 01:55:06.882365 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:06.882328 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:55:06.882816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:06.882788 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:55:06.883669 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:06.883648 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:55:17.630759 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:55:17.630718 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:55:32.767405 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:32.767367 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7"] Oct 30 01:55:32.803673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:32.803625 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc"] Oct 30 01:55:37.768495 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:37.768432 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" podUID="741f058f-d2ee-40d8-9f9d-6fac9c19e3ae" containerName="payload-processing" containerID="cri-o://a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce" gracePeriod=25 Oct 30 01:55:37.805182 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:37.805128 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" podUID="cdd4d3b9-abfd-4371-9c89-d1badb083c78" containerName="payload-pre-processing" containerID="cri-o://89d4cf7bff88e54f80e51d9fa2ae0a55c37e454f08598e18bb6903d1a42bcf9b" gracePeriod=25 Oct 30 01:55:38.136249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.136214 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:55:38.312822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.312413 5035 generic.go:356] "Generic (PLEG): container finished" podID="cdd4d3b9-abfd-4371-9c89-d1badb083c78" containerID="89d4cf7bff88e54f80e51d9fa2ae0a55c37e454f08598e18bb6903d1a42bcf9b" exitCode=0 Oct 30 01:55:38.312822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.312705 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" event={"ID":"cdd4d3b9-abfd-4371-9c89-d1badb083c78","Type":"ContainerDied","Data":"89d4cf7bff88e54f80e51d9fa2ae0a55c37e454f08598e18bb6903d1a42bcf9b"} Oct 30 01:55:38.312822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.312769 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" event={"ID":"cdd4d3b9-abfd-4371-9c89-d1badb083c78","Type":"ContainerDied","Data":"0f19811203ebb0780794c80a744f42a36dab8386daa5a9751c1685209ecb5603"} Oct 30 01:55:38.312822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.312794 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f19811203ebb0780794c80a744f42a36dab8386daa5a9751c1685209ecb5603" Oct 30 01:55:38.315679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.313500 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:55:38.315679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.314358 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-plugins-config-volume\") pod \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\" (UID: \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\") " Oct 30 01:55:38.315679 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.314468 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-kube-api-access-spc78\" (UniqueName: \"kubernetes.io/projected/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-kube-api-access-spc78\") pod \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\" (UID: \"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae\") " Oct 30 01:55:38.316103 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.316016 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-plugins-config-volume" pod "741f058f-d2ee-40d8-9f9d-6fac9c19e3ae" (UID: "741f058f-d2ee-40d8-9f9d-6fac9c19e3ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:38.320352 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.319721 5035 generic.go:356] "Generic (PLEG): container finished" podID="741f058f-d2ee-40d8-9f9d-6fac9c19e3ae" containerID="a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce" exitCode=0 Oct 30 01:55:38.320352 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.319773 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" event={"ID":"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae","Type":"ContainerDied","Data":"a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce"} Oct 30 01:55:38.320352 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.319829 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" event={"ID":"741f058f-d2ee-40d8-9f9d-6fac9c19e3ae","Type":"ContainerDied","Data":"12d05b47e0f0cc314e2ae91b05e74adbadcc8c5216227a0a703262db2b529f84"} Oct 30 01:55:38.320352 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.319861 5035 scope.go:122] "RemoveContainer" containerID="a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce" Oct 30 01:55:38.320352 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.320045 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7" Oct 30 01:55:38.326738 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.326655 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-kube-api-access-spc78" pod "741f058f-d2ee-40d8-9f9d-6fac9c19e3ae" (UID: "741f058f-d2ee-40d8-9f9d-6fac9c19e3ae"). InnerVolumeSpecName "kube-api-access-spc78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:38.350885 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.347663 5035 scope.go:122] "RemoveContainer" containerID="a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce" Oct 30 01:55:38.357335 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:55:38.357102 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce\": container with ID starting with a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce not found: ID does not exist" containerID="a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce" Oct 30 01:55:38.357335 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.357175 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce"} err="failed to get container status \"a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce\": rpc error: code = NotFound desc = could not find container \"a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce\": container with ID starting with a8aeaaa4af35c97f3df744311af064b3c89806f8773a13818b0d19e9484455ce not found: ID does not exist" Oct 30 01:55:38.416611 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.416207 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd4d3b9-abfd-4371-9c89-d1badb083c78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d3b9-abfd-4371-9c89-d1badb083c78-plugins-config-volume\") pod \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\" (UID: \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\") " Oct 30 01:55:38.416611 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.416473 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd4d3b9-abfd-4371-9c89-d1badb083c78-kube-api-access-dmpxb\" (UniqueName: \"kubernetes.io/projected/cdd4d3b9-abfd-4371-9c89-d1badb083c78-kube-api-access-dmpxb\") pod \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\" (UID: \"cdd4d3b9-abfd-4371-9c89-d1badb083c78\") " Oct 30 01:55:38.416891 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.416807 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd4d3b9-abfd-4371-9c89-d1badb083c78-plugins-config-volume" pod "cdd4d3b9-abfd-4371-9c89-d1badb083c78" (UID: "cdd4d3b9-abfd-4371-9c89-d1badb083c78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:38.417280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.417254 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spc78\" (UniqueName: \"kubernetes.io/projected/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-kube-api-access-spc78\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:38.417392 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.417284 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd4d3b9-abfd-4371-9c89-d1badb083c78-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:38.417392 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.417301 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:38.419570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.419505 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd4d3b9-abfd-4371-9c89-d1badb083c78-kube-api-access-dmpxb" pod "cdd4d3b9-abfd-4371-9c89-d1badb083c78" (UID: "cdd4d3b9-abfd-4371-9c89-d1badb083c78"). InnerVolumeSpecName "kube-api-access-dmpxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:38.519167 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.519117 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmpxb\" (UniqueName: \"kubernetes.io/projected/cdd4d3b9-abfd-4371-9c89-d1badb083c78-kube-api-access-dmpxb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:38.672197 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.672120 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7"] Oct 30 01:55:38.685081 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:38.685043 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2198e4-585bd74748-5sfg7"] Oct 30 01:55:39.328119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:39.328088 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc" Oct 30 01:55:39.385968 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:39.385927 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc"] Oct 30 01:55:39.398685 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:39.398648 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2198e4-7ffb8866c9-qrbgc"] Oct 30 01:55:39.646188 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:39.646102 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741f058f-d2ee-40d8-9f9d-6fac9c19e3ae" path="/var/lib/kubelet/pods/741f058f-d2ee-40d8-9f9d-6fac9c19e3ae/volumes" Oct 30 01:55:39.646804 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:39.646784 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd4d3b9-abfd-4371-9c89-d1badb083c78" path="/var/lib/kubelet/pods/cdd4d3b9-abfd-4371-9c89-d1badb083c78/volumes" Oct 30 01:55:44.551290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.551236 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv"] Oct 30 01:55:44.552930 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.552895 5035 cpu_manager.go:414] "containerMap: removing container" podUID="741f058f-d2ee-40d8-9f9d-6fac9c19e3ae" containerName="payload-processing" Oct 30 01:55:44.552930 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.552929 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741f058f-d2ee-40d8-9f9d-6fac9c19e3ae" containerName="payload-processing" Oct 30 01:55:44.553178 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.552957 5035 cpu_manager.go:414] "containerMap: removing container" podUID="cdd4d3b9-abfd-4371-9c89-d1badb083c78" containerName="payload-pre-processing" Oct 30 01:55:44.553178 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.552966 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd4d3b9-abfd-4371-9c89-d1badb083c78" containerName="payload-pre-processing" Oct 30 01:55:44.553338 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.553319 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="741f058f-d2ee-40d8-9f9d-6fac9c19e3ae" containerName="payload-processing" Oct 30 01:55:44.553388 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.553352 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd4d3b9-abfd-4371-9c89-d1badb083c78" containerName="payload-pre-processing" Oct 30 01:55:44.561588 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.561554 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.563597 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.563565 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp"] Oct 30 01:55:44.567958 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.567929 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.582816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.582782 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv"] Oct 30 01:55:44.607163 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.607125 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp"] Oct 30 01:55:44.718487 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.718436 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.718716 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.718505 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkpc\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-kube-api-access-ntkpc\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.718716 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.718626 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.718716 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.718683 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.718716 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.718709 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aadb65e-4b07-40c9-8711-b99d41248123-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.718856 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.718758 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2948\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-kube-api-access-q2948\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.718904 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.718879 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.719010 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.718995 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.719061 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719047 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.719097 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719086 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aadb65e-4b07-40c9-8711-b99d41248123-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.719133 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719113 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.719167 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719145 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.719220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719206 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.719255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719234 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.719296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719283 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.719333 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719310 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.719364 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719354 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.719476 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.719463 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.823335 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.823200 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.823710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.823678 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.823710 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.823698 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.823913 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.823770 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.823913 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.823829 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkpc\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-kube-api-access-ntkpc\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.823913 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.823855 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.824056 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.823929 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.824056 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.823955 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aadb65e-4b07-40c9-8711-b99d41248123-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.824056 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.823978 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2948\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-kube-api-access-q2948\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.824056 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824016 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.824056 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824040 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.824255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824181 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.824255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824231 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.824370 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824305 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aadb65e-4b07-40c9-8711-b99d41248123-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.824370 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824336 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.824460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824363 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.824460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824416 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.824572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824509 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.824572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824558 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.824683 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824585 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.824683 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.824608 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.825321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.825298 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.825907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.825882 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.827601 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.827579 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aadb65e-4b07-40c9-8711-b99d41248123-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.828762 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.828738 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.830154 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.829432 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.830154 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.829626 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.830154 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.829913 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.830154 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.830121 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aadb65e-4b07-40c9-8711-b99d41248123-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.830417 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.830392 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.831051 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.831030 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.831154 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.831056 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.849294 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.849253 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.849451 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.849419 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkpc\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-kube-api-access-ntkpc\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.850411 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.850389 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2948\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-kube-api-access-q2948\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.852658 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.852634 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:44.881846 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.881800 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:44.890832 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:44.890797 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:45.309945 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:45.309907 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv"] Oct 30 01:55:45.315670 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:55:45.315633 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aadb65e_4b07_40c9_8711_b99d41248123.slice/crio-69ccaabb63f4212cfef15ff74493dbd5de22ee670d6ef338287ba1aa562e3374 WatchSource:0}: Error finding container 69ccaabb63f4212cfef15ff74493dbd5de22ee670d6ef338287ba1aa562e3374: Status 404 returned error can't find the container with id 69ccaabb63f4212cfef15ff74493dbd5de22ee670d6ef338287ba1aa562e3374 Oct 30 01:55:45.318487 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:45.318452 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:45.318594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:45.318546 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:45.318594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:45.318587 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:45.376303 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:45.376273 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" event={"ID":"8aadb65e-4b07-40c9-8711-b99d41248123","Type":"ContainerStarted","Data":"69ccaabb63f4212cfef15ff74493dbd5de22ee670d6ef338287ba1aa562e3374"} Oct 30 01:55:45.475664 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:55:45.475617 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e35d96_3162_49f8_bfa0_070f1f4da6d6.slice/crio-d4142ff86b15ea0ffa52d5bfb52c92d6a4cbe521ce1e9167d454638c6efc89eb WatchSource:0}: Error finding container d4142ff86b15ea0ffa52d5bfb52c92d6a4cbe521ce1e9167d454638c6efc89eb: Status 404 returned error can't find the container with id d4142ff86b15ea0ffa52d5bfb52c92d6a4cbe521ce1e9167d454638c6efc89eb Oct 30 01:55:45.476736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:45.476709 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp"] Oct 30 01:55:45.479422 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:45.479379 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:45.479596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:45.479476 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:45.479596 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:45.479548 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:46.424367 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.424330 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" event={"ID":"8aadb65e-4b07-40c9-8711-b99d41248123","Type":"ContainerStarted","Data":"b05fa652a5d39e8c4336cd073f4f60a218a1f7763164130d73c786021e6f46ec"} Oct 30 01:55:46.427278 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.427248 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" event={"ID":"23e35d96-3162-49f8-bfa0-070f1f4da6d6","Type":"ContainerStarted","Data":"82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3"} Oct 30 01:55:46.427427 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.427297 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" event={"ID":"23e35d96-3162-49f8-bfa0-070f1f4da6d6","Type":"ContainerStarted","Data":"d4142ff86b15ea0ffa52d5bfb52c92d6a4cbe521ce1e9167d454638c6efc89eb"} Oct 30 01:55:46.623362 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.623296 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podStartSLOduration=2.623277849 podStartE2EDuration="2.623277849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:46.615929094 +0000 UTC m=+1625.118861317" watchObservedRunningTime="2026-10-30 01:55:46.623277849 +0000 UTC m=+1625.126210068" Oct 30 01:55:46.624926 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.624872 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podStartSLOduration=2.624861328 podStartE2EDuration="2.624861328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:46.523675542 +0000 UTC m=+1625.026607783" watchObservedRunningTime="2026-10-30 01:55:46.624861328 +0000 UTC m=+1625.127793548" Oct 30 01:55:46.892555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.888943 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:46.892555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.889118 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:55:46.892555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.889174 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:55:46.895573 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.895443 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:46.895773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.895644 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:55:46.895773 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:46.895691 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:55:47.655156 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:55:47.655125 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:55:47.883024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:47.882982 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:55:47.883284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:47.883256 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:55:47.903026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:47.902985 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:55:47.904397 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:47.904359 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:55:48.887396 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:48.887357 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:55:48.888113 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:48.888076 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:55:48.900059 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:48.900017 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:55:48.900379 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:48.900344 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:55:49.882950 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:49.882908 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:55:49.883790 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:49.883273 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:55:49.892137 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:49.892093 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:55:49.892762 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:49.892730 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:55:50.457055 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:50.457007 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:50.949725 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:50.941310 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:55:50.949725 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:50.941389 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:55:51.036166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.008241 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.036166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.009564 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Oct 30 01:55:51.061330 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.036644 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 01:55:51.142685 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.064154 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:55:51.142685 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.064220 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:55:51.183651 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.183603 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8"] Oct 30 01:55:51.219762 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.219638 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.263827 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.263789 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e81ef2c-nc8h8"] Oct 30 01:55:51.458550 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.457495 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.458550 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.458204 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:55:51.960550 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.959284 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:55:51.960550 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:51.959344 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:55:52.033598 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.009427 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.035549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.035495 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:55:52.036071 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.035951 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:55:52.036925 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.036806 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.047969 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.047929 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da600374-f947-4104-906d-aa030d698e51" path="/var/lib/kubelet/pods/da600374-f947-4104-906d-aa030d698e51/volumes" Oct 30 01:55:52.106559 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.083210 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7"] Oct 30 01:55:52.108331 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.108295 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:55:52.135637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.108767 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerName="istio-proxy" containerID="cri-o://1c697babf527ef5348155efcf8dee39c95288fa1cbd9f9f1884beedb4c606b4f" gracePeriod=30 Oct 30 01:55:52.313617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.276731 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.313617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.277288 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:55:52.538554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.389104 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.538554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.458467 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.807763 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:52.807712 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.054900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.054852 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 01:55:53.356568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.055603 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" podUID="ed9fd4ef-0aab-454d-b95f-83cbcb4fe367" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 30 01:55:53.356568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.056783 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.356568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.354542 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:55:53.366071 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.149701 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" podUID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.366384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.293103 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.366589 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.187950 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:55:53.366731 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.366699 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:55:53.366850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.188083 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:55:53.366949 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.366934 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:55:53.384335 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.372562 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 01:55:53.433473 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.433422 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.443612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.438243 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:55:53.500714 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.500663 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 01:55:53.822217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:53.821401 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:55:54.110986 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.110876 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-dw8g7" podUID="dca83999-a27c-4d5b-8ab5-09ccbf01f5a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 01:55:54.169640 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.169592 5035 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:54.170450 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.170422 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:55:54.170643 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.170620 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:55:54.174554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.173808 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:55:54.174554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.173877 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:55:54.327723 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.327681 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:55:54.327941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.327759 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 01:55:54.327941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.327795 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 01:55:54.327941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.327831 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 01:55:54.327941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.327866 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj" Oct 30 01:55:54.333820 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.333787 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 01:55:54.371234 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:54.371133 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 01:55:55.924176 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:55.923743 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:55:55.924176 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:55.923815 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:55.932717 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:55.932678 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:55:55.932942 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:55.932917 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:56.548901 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:56.548864 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:56.549399 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:56.549360 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:56.550145 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:56.550124 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 01:55:56.783789 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:56.783754 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:56.784217 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:56.784197 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:56.785089 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:56.785067 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 01:55:57.356063 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.356017 5035 generic.go:356] "Generic (PLEG): container finished" podID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerID="1c697babf527ef5348155efcf8dee39c95288fa1cbd9f9f1884beedb4c606b4f" exitCode=0 Oct 30 01:55:57.356517 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.356067 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" event={"ID":"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7","Type":"ContainerDied","Data":"1c697babf527ef5348155efcf8dee39c95288fa1cbd9f9f1884beedb4c606b4f"} Oct 30 01:55:57.496834 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.496762 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:55:57.628936 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.628901 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istiod-ca-cert\") pod \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " Oct 30 01:55:57.629127 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.628992 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-certs\") pod \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " Oct 30 01:55:57.629167 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629153 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-kube-api-access-zvmff\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-kube-api-access-zvmff\") pod \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " Oct 30 01:55:57.629242 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629222 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-credential-socket\") pod \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " Oct 30 01:55:57.629336 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629305 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istiod-ca-cert" pod "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" (UID: "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:57.629336 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629325 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-certs" pod "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" (UID: "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.629458 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629316 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-data\") pod \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " Oct 30 01:55:57.629458 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629405 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-podinfo\") pod \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " Oct 30 01:55:57.629597 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629470 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-socket\") pod \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " Oct 30 01:55:57.629597 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629471 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-credential-socket" pod "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" (UID: "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.629597 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629551 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-data" pod "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" (UID: "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.629723 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629594 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-envoy\") pod \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " Oct 30 01:55:57.629723 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629663 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-token\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-token\") pod \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\" (UID: \"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7\") " Oct 30 01:55:57.629723 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.629682 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-socket" pod "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" (UID: "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.630475 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.630447 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.630475 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.630477 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.630775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.630490 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.630775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.630505 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.630775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.630518 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.631796 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.631769 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-kube-api-access-zvmff" pod "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" (UID: "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7"). InnerVolumeSpecName "kube-api-access-zvmff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:57.631964 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.631937 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-podinfo" pod "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" (UID: "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:55:57.632165 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.632137 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-envoy" pod "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" (UID: "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.632280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.632265 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-token" pod "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" (UID: "7c6d4a1f-957e-47d8-bad7-81b9b68af2e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:57.731523 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.731466 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.731523 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.731519 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.731783 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.731573 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvmff\" (UniqueName: \"kubernetes.io/projected/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-kube-api-access-zvmff\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.731783 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:57.731589 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:55:58.365634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.365593 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" event={"ID":"7c6d4a1f-957e-47d8-bad7-81b9b68af2e7","Type":"ContainerDied","Data":"185f2aace711f1ab952cfaed7cb7fc8d28ea6c40bca90c34e9d3f5998f7e2366"} Oct 30 01:55:58.366125 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.365666 5035 scope.go:122] "RemoveContainer" containerID="1c697babf527ef5348155efcf8dee39c95288fa1cbd9f9f1884beedb4c606b4f" Oct 30 01:55:58.366125 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.365609 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7" Oct 30 01:55:58.433456 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.433416 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7"] Oct 30 01:55:58.446547 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.446486 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-5k2f7"] Oct 30 01:55:58.673471 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.673383 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5"] Oct 30 01:55:58.674108 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.674092 5035 cpu_manager.go:414] "containerMap: removing container" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerName="istio-proxy" Oct 30 01:55:58.674157 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.674109 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerName="istio-proxy" Oct 30 01:55:58.674270 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.674260 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" containerName="istio-proxy" Oct 30 01:55:58.679213 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.679181 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.697070 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.697035 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2933ff00-openshift-default-dockercfg-h4nbj\"" Oct 30 01:55:58.704384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.704334 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5"] Oct 30 01:55:58.847062 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.846994 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-token\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.847267 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.847093 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-envoy\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.847267 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.847116 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-podinfo\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.847267 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.847143 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-credential-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.847440 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.847312 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-data\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.847440 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.847371 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istiod-ca-cert\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.847593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.847443 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-certs\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.847660 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.847623 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.847755 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.847738 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-kube-api-access-l68bs\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.949367 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949266 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istiod-ca-cert\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.949367 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949334 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-certs\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.949684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949396 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.949684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949456 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-kube-api-access-l68bs\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.949684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949510 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-token\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.949684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949580 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-envoy\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.949684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949599 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-podinfo\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.949684 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949626 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-credential-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.950003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949718 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-data\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.950003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949861 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-certs\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.950003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.949900 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.950157 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.950032 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istiod-ca-cert\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.950157 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.950062 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-data\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.950243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.950219 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-credential-socket\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.952278 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.952251 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-envoy\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.952486 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.952469 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-podinfo\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.976398 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.976369 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-token\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.976600 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.976511 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-kube-api-access-l68bs\") pod \"e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:58.994187 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:58.994153 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:55:59.348811 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:59.348165 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5"] Oct 30 01:55:59.357126 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:59.357065 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:59.357406 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:59.357380 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:59.357636 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:59.357606 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:55:59.379370 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:59.379327 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" event={"ID":"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f","Type":"ContainerStarted","Data":"6ba5d3e4ff48517d94d7820a0ce69d6ec6c6e5f55228188ced780e499b8e399d"} Oct 30 01:55:59.645614 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:55:59.645512 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6d4a1f-957e-47d8-bad7-81b9b68af2e7" path="/var/lib/kubelet/pods/7c6d4a1f-957e-47d8-bad7-81b9b68af2e7/volumes" Oct 30 01:56:00.407758 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:00.407722 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" event={"ID":"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f","Type":"ContainerStarted","Data":"678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a"} Oct 30 01:56:00.493622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:00.492737 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" podStartSLOduration=2.492712299 podStartE2EDuration="2.492712299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:00.488513313 +0000 UTC m=+1638.991445534" watchObservedRunningTime="2026-10-30 01:56:00.492712299 +0000 UTC m=+1638.995644548" Oct 30 01:56:01.006563 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:01.004109 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:56:01.026843 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:01.026805 5035 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:56:01.056063 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:01.034839 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:56:01.997556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:01.995227 5035 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:56:01.997556 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:01.995288 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:56:02.995228 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:02.995183 5035 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:56:02.995453 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:02.995263 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:56:03.037775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:03.037737 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n"] Oct 30 01:56:03.050280 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:03.050241 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb3fdffa-kb28n"] Oct 30 01:56:03.645429 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:03.645396 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c463943-87e6-4dfc-b83b-7ca69301fd35" path="/var/lib/kubelet/pods/9c463943-87e6-4dfc-b83b-7ca69301fd35/volumes" Oct 30 01:56:03.995296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:03.995246 5035 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:56:03.995572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:03.995351 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:56:04.994944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:04.994889 5035 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:56:04.995472 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:04.994962 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:56:05.999489 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:05.999453 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:56:06.000032 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:06.000007 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:56:06.000877 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:06.000852 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:56:16.970728 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:16.970687 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn"] Oct 30 01:56:16.976889 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:16.976855 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.044271 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.044187 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn"] Oct 30 01:56:17.044735 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.044711 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-922b2c-openshift-default-dockercfg-p78gv\"" Oct 30 01:56:17.057128 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.057087 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.057318 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.057152 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.057318 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.057200 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.057507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.057402 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.057675 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.057516 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.057675 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.057639 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmth\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-kube-api-access-dwmth\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.057799 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.057766 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.057866 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.057846 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37613cf-b692-4661-8206-a3385fd1c2f4-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.058013 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.057990 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.159900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.159850 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.159913 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.159978 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmth\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-kube-api-access-dwmth\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.160067 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160258 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.160200 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37613cf-b692-4661-8206-a3385fd1c2f4-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.160261 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.160297 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160424 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.160399 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160477 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.160440 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160780 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.160481 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.160780 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.160608 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.161571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.160932 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.161571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.161030 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.161571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.161056 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37613cf-b692-4661-8206-a3385fd1c2f4-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.162840 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.162820 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.162944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.162859 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.191448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.191407 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.191842 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.191822 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmth\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-kube-api-access-dwmth\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.337252 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.337152 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:17.631071 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:56:17.630984 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:56:17.697863 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.697832 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn"] Oct 30 01:56:17.697996 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:56:17.697868 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37613cf_b692_4661_8206_a3385fd1c2f4.slice/crio-e53258ae75b897b87776040ef8f7ab4e2fb186315ed8d5a865cca6a591d0d296 WatchSource:0}: Error finding container e53258ae75b897b87776040ef8f7ab4e2fb186315ed8d5a865cca6a591d0d296: Status 404 returned error can't find the container with id e53258ae75b897b87776040ef8f7ab4e2fb186315ed8d5a865cca6a591d0d296 Oct 30 01:56:17.700839 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.700800 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:56:17.700940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.700871 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:56:17.700940 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:17.700900 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:56:18.652379 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:18.652337 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" event={"ID":"e37613cf-b692-4661-8206-a3385fd1c2f4","Type":"ContainerStarted","Data":"7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635"} Oct 30 01:56:18.652833 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:18.652397 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" event={"ID":"e37613cf-b692-4661-8206-a3385fd1c2f4","Type":"ContainerStarted","Data":"e53258ae75b897b87776040ef8f7ab4e2fb186315ed8d5a865cca6a591d0d296"} Oct 30 01:56:18.760954 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:18.760892 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" podStartSLOduration=2.760877754 podStartE2EDuration="2.760877754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:18.758885059 +0000 UTC m=+1657.261817279" watchObservedRunningTime="2026-10-30 01:56:18.760877754 +0000 UTC m=+1657.263809973" Oct 30 01:56:19.337938 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:19.337898 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:19.340221 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:19.340179 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:56:19.340422 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:19.340248 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 01:56:20.337981 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:20.337941 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:56:20.338442 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:20.338000 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 01:56:21.338319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:21.338276 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:56:21.338775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:21.338341 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 01:56:22.036470 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:22.036428 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn"] Oct 30 01:56:22.049617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:22.049576 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6816c1-pgltn"] Oct 30 01:56:22.337703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:22.337605 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:56:22.337703 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:22.337678 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 01:56:23.338047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:23.337948 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:56:23.338047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:23.338034 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 01:56:23.645768 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:23.645683 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fc9164-4fa4-4468-8b0c-f301b1c91f4e" path="/var/lib/kubelet/pods/93fc9164-4fa4-4468-8b0c-f301b1c91f4e/volumes" Oct 30 01:56:24.344920 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.344890 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:24.346213 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.346185 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:24.348511 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.348487 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:56:24.508458 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.508413 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr"] Oct 30 01:56:24.516865 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.516830 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:24.532931 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.532895 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr"] Oct 30 01:56:24.533411 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.533383 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b2c11f-dockercfg-6fjt8\"" Oct 30 01:56:24.533602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.533493 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b2c11f\"" Oct 30 01:56:24.631405 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.631272 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f89f71-2cbe-4483-a4e0-78723415f1f9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr\" (UID: \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:24.632119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.632087 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxfp\" (UniqueName: \"kubernetes.io/projected/c5f89f71-2cbe-4483-a4e0-78723415f1f9-kube-api-access-qxxfp\") pod \"payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr\" (UID: \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:24.733515 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.733470 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxfp\" (UniqueName: \"kubernetes.io/projected/c5f89f71-2cbe-4483-a4e0-78723415f1f9-kube-api-access-qxxfp\") pod \"payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr\" (UID: \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:24.733753 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.733577 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f89f71-2cbe-4483-a4e0-78723415f1f9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr\" (UID: \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:24.734605 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.734550 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f89f71-2cbe-4483-a4e0-78723415f1f9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr\" (UID: \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:24.771134 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.771098 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxfp\" (UniqueName: \"kubernetes.io/projected/c5f89f71-2cbe-4483-a4e0-78723415f1f9-kube-api-access-qxxfp\") pod \"payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr\" (UID: \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:24.838965 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:24.838924 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:25.176827 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:25.176793 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr"] Oct 30 01:56:25.177156 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:56:25.177130 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f89f71_2cbe_4483_a4e0_78723415f1f9.slice/crio-ea6769fc92deb5fb6558b4627277e5c6ed299cc2a8fb0eacd21ffd78b29e7162 WatchSource:0}: Error finding container ea6769fc92deb5fb6558b4627277e5c6ed299cc2a8fb0eacd21ffd78b29e7162: Status 404 returned error can't find the container with id ea6769fc92deb5fb6558b4627277e5c6ed299cc2a8fb0eacd21ffd78b29e7162 Oct 30 01:56:25.776376 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:25.776331 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" event={"ID":"c5f89f71-2cbe-4483-a4e0-78723415f1f9","Type":"ContainerStarted","Data":"e3f94ec15a68b44e47ff4275604450beaf100c8bf07d0bad58b992b6ea45a92b"} Oct 30 01:56:25.776833 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:25.776432 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:25.776833 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:25.776459 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" event={"ID":"c5f89f71-2cbe-4483-a4e0-78723415f1f9","Type":"ContainerStarted","Data":"ea6769fc92deb5fb6558b4627277e5c6ed299cc2a8fb0eacd21ffd78b29e7162"} Oct 30 01:56:25.821974 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:25.821910 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" podStartSLOduration=1.8218939870000002 podStartE2EDuration="1.821893987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:25.819595753 +0000 UTC m=+1664.322528060" watchObservedRunningTime="2026-10-30 01:56:25.821893987 +0000 UTC m=+1664.324826238" Oct 30 01:56:26.814296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:26.814256 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v"] Oct 30 01:56:31.815245 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:31.815168 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" podUID="fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" containerName="payload-pre-processing" containerID="cri-o://b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088" gracePeriod=25 Oct 30 01:56:35.317791 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:35.317742 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gjlm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:56:35.318499 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:35.317844 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:36.787746 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:36.787702 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:56:41.218347 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:41.218308 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" podUID="fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:56:47.632623 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:56:47.632567 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:56:51.219673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:51.219609 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" podUID="fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:56:52.111276 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.111241 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:56:52.193813 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.193759 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-plugins-config-volume\") pod \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\" (UID: \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\") " Oct 30 01:56:52.194037 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.193873 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-kube-api-access-dm4m4\" (UniqueName: \"kubernetes.io/projected/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-kube-api-access-dm4m4\") pod \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\" (UID: \"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8\") " Oct 30 01:56:52.194229 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.194205 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-plugins-config-volume" pod "fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" (UID: "fc4dd90b-5e24-44aa-a3a3-0369d58f65e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:52.194560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.194543 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:56:52.196154 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.196124 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-kube-api-access-dm4m4" pod "fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" (UID: "fc4dd90b-5e24-44aa-a3a3-0369d58f65e8"). InnerVolumeSpecName "kube-api-access-dm4m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:52.278112 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.278070 5035 generic.go:356] "Generic (PLEG): container finished" podID="fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" containerID="b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088" exitCode=0 Oct 30 01:56:52.278609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.278125 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" Oct 30 01:56:52.278609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.278153 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" event={"ID":"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8","Type":"ContainerDied","Data":"b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088"} Oct 30 01:56:52.278609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.278219 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v" event={"ID":"fc4dd90b-5e24-44aa-a3a3-0369d58f65e8","Type":"ContainerDied","Data":"27a45658a156e41d8ec8f48faba0521f7877e85544335ab28bef5da41f53fe3e"} Oct 30 01:56:52.278609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.278246 5035 scope.go:122] "RemoveContainer" containerID="b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088" Oct 30 01:56:52.290284 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.290259 5035 scope.go:122] "RemoveContainer" containerID="b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088" Oct 30 01:56:52.290649 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:56:52.290624 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088\": container with ID starting with b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088 not found: ID does not exist" containerID="b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088" Oct 30 01:56:52.290728 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.290658 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088"} err="failed to get container status \"b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088\": rpc error: code = NotFound desc = could not find container \"b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088\": container with ID starting with b1a29a66f6dee0a291fe48c7aee4e2bb64b40206d4417b356039a48eb2656088 not found: ID does not exist" Oct 30 01:56:52.295734 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.295706 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4m4\" (UniqueName: \"kubernetes.io/projected/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8-kube-api-access-dm4m4\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:56:52.374631 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.374539 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v"] Oct 30 01:56:52.385930 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:52.385896 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-6h77v"] Oct 30 01:56:53.654549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:53.654497 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" path="/var/lib/kubelet/pods/fc4dd90b-5e24-44aa-a3a3-0369d58f65e8/volumes" Oct 30 01:56:58.254100 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.254051 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb"] Oct 30 01:56:58.254909 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.254865 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" podUID="6a6ffb31-b827-408e-b303-7b3060a17d1f" containerName="maas-api" containerID="cri-o://4c006412f4f8e4603fb840b59bdd843b8602a0f1c723701ad05bb7c6be762374" gracePeriod=30 Oct 30 01:56:58.382896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.382862 5035 generic.go:356] "Generic (PLEG): container finished" podID="6a6ffb31-b827-408e-b303-7b3060a17d1f" containerID="4c006412f4f8e4603fb840b59bdd843b8602a0f1c723701ad05bb7c6be762374" exitCode=0 Oct 30 01:56:58.383043 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.382934 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" event={"ID":"6a6ffb31-b827-408e-b303-7b3060a17d1f","Type":"ContainerDied","Data":"4c006412f4f8e4603fb840b59bdd843b8602a0f1c723701ad05bb7c6be762374"} Oct 30 01:56:58.630504 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.630150 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:56:58.716708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.714515 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-maas-api-tls\") pod \"6a6ffb31-b827-408e-b303-7b3060a17d1f\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " Oct 30 01:56:58.716708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.714648 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6ffb31-b827-408e-b303-7b3060a17d1f-kube-api-access-vn85j\" (UniqueName: \"kubernetes.io/projected/6a6ffb31-b827-408e-b303-7b3060a17d1f-kube-api-access-vn85j\") pod \"6a6ffb31-b827-408e-b303-7b3060a17d1f\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " Oct 30 01:56:58.716708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.714743 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-metrics-certs\") pod \"6a6ffb31-b827-408e-b303-7b3060a17d1f\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " Oct 30 01:56:58.716708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.714927 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6ffb31-b827-408e-b303-7b3060a17d1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6ffb31-b827-408e-b303-7b3060a17d1f-openshift-service-ca\") pod \"6a6ffb31-b827-408e-b303-7b3060a17d1f\" (UID: \"6a6ffb31-b827-408e-b303-7b3060a17d1f\") " Oct 30 01:56:58.718018 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.717663 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6ffb31-b827-408e-b303-7b3060a17d1f-openshift-service-ca" pod "6a6ffb31-b827-408e-b303-7b3060a17d1f" (UID: "6a6ffb31-b827-408e-b303-7b3060a17d1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:58.726238 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.726185 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-metrics-certs" pod "6a6ffb31-b827-408e-b303-7b3060a17d1f" (UID: "6a6ffb31-b827-408e-b303-7b3060a17d1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:58.730006 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.729948 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-maas-api-tls" pod "6a6ffb31-b827-408e-b303-7b3060a17d1f" (UID: "6a6ffb31-b827-408e-b303-7b3060a17d1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:58.735150 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.734699 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6ffb31-b827-408e-b303-7b3060a17d1f-kube-api-access-vn85j" pod "6a6ffb31-b827-408e-b303-7b3060a17d1f" (UID: "6a6ffb31-b827-408e-b303-7b3060a17d1f"). InnerVolumeSpecName "kube-api-access-vn85j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:58.828612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.817502 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6ffb31-b827-408e-b303-7b3060a17d1f-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:56:58.828612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.817614 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:56:58.828612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.817626 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn85j\" (UniqueName: \"kubernetes.io/projected/6a6ffb31-b827-408e-b303-7b3060a17d1f-kube-api-access-vn85j\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:56:58.828612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:58.817635 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6ffb31-b827-408e-b303-7b3060a17d1f-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:56:59.394437 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:59.394402 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" Oct 30 01:56:59.394437 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:59.394415 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb" event={"ID":"6a6ffb31-b827-408e-b303-7b3060a17d1f","Type":"ContainerDied","Data":"84ea1696b4f61dea73c29c9937475fe7266cbe42b414ce410ab06ddf4d7595bb"} Oct 30 01:56:59.394975 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:59.394475 5035 scope.go:122] "RemoveContainer" containerID="4c006412f4f8e4603fb840b59bdd843b8602a0f1c723701ad05bb7c6be762374" Oct 30 01:56:59.451794 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:59.451756 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb"] Oct 30 01:56:59.464613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:59.464576 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2933ff00-5cc657f648-sxpzb"] Oct 30 01:56:59.644047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:56:59.644009 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6ffb31-b827-408e-b303-7b3060a17d1f" path="/var/lib/kubelet/pods/6a6ffb31-b827-408e-b303-7b3060a17d1f/volumes" Oct 30 01:57:02.082469 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.082433 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl"] Oct 30 01:57:02.083339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.083310 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6a6ffb31-b827-408e-b303-7b3060a17d1f" containerName="maas-api" Oct 30 01:57:02.083339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.083338 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6ffb31-b827-408e-b303-7b3060a17d1f" containerName="maas-api" Oct 30 01:57:02.083463 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.083389 5035 cpu_manager.go:414] "containerMap: removing container" podUID="fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" containerName="payload-pre-processing" Oct 30 01:57:02.083463 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.083395 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" containerName="payload-pre-processing" Oct 30 01:57:02.083568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.083505 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6ffb31-b827-408e-b303-7b3060a17d1f" containerName="maas-api" Oct 30 01:57:02.083568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.083521 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4dd90b-5e24-44aa-a3a3-0369d58f65e8" containerName="payload-pre-processing" Oct 30 01:57:02.089113 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.089084 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.115649 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.115587 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl"] Oct 30 01:57:02.175302 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.175254 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.175479 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.175372 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.175562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.175510 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.175700 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.175678 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.175753 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.175738 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdf5h\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-kube-api-access-bdf5h\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.175860 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.175842 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27114de7-baaf-4f00-aef5-146ea1b7d580-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.175907 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.175886 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.175943 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.175914 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.176004 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.175992 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.277754 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.277716 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.277975 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.277832 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.277975 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.277855 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.277975 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.277919 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278035 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278100 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdf5h\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-kube-api-access-bdf5h\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278238 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278154 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27114de7-baaf-4f00-aef5-146ea1b7d580-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278238 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278198 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278238 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278208 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278238 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278224 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278458 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278269 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278458 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278327 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278642 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278617 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.278906 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.278886 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27114de7-baaf-4f00-aef5-146ea1b7d580-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.280673 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.280649 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.280795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.280752 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.302976 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.302944 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.303256 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.303230 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdf5h\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-kube-api-access-bdf5h\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.404464 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.404360 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:02.738784 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.738748 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl"] Oct 30 01:57:02.739033 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:57:02.739007 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27114de7_baaf_4f00_aef5_146ea1b7d580.slice/crio-55d503a766186b4b70c6239431ec4a6c6e79c0bc05607a0ad0b959f255775f71 WatchSource:0}: Error finding container 55d503a766186b4b70c6239431ec4a6c6e79c0bc05607a0ad0b959f255775f71: Status 404 returned error can't find the container with id 55d503a766186b4b70c6239431ec4a6c6e79c0bc05607a0ad0b959f255775f71 Oct 30 01:57:02.741813 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.741779 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:57:02.741903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.741848 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:57:02.741903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:02.741878 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:57:03.444354 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.444305 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" event={"ID":"27114de7-baaf-4f00-aef5-146ea1b7d580","Type":"ContainerStarted","Data":"a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540"} Oct 30 01:57:03.444985 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.444941 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" event={"ID":"27114de7-baaf-4f00-aef5-146ea1b7d580","Type":"ContainerStarted","Data":"55d503a766186b4b70c6239431ec4a6c6e79c0bc05607a0ad0b959f255775f71"} Oct 30 01:57:03.532466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.532396 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" podStartSLOduration=1.532375868 podStartE2EDuration="1.532375868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:03.517257633 +0000 UTC m=+1702.020189854" watchObservedRunningTime="2026-10-30 01:57:03.532375868 +0000 UTC m=+1702.035308088" Oct 30 01:57:03.618404 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.618355 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67"] Oct 30 01:57:03.626719 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.626683 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:03.652478 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.652275 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2818a6\"" Oct 30 01:57:03.652478 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.652443 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2818a6-dockercfg-jd6dl\"" Oct 30 01:57:03.779667 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.776028 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxx4\" (UniqueName: \"kubernetes.io/projected/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-kube-api-access-jqxx4\") pod \"payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67\" (UID: \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:03.779667 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.776126 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67\" (UID: \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:03.814187 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.814144 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67"] Oct 30 01:57:03.882937 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.882883 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67\" (UID: \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:03.883680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.883645 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxx4\" (UniqueName: \"kubernetes.io/projected/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-kube-api-access-jqxx4\") pod \"payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67\" (UID: \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:03.908631 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:03.885393 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67\" (UID: \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:04.077388 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:04.077244 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxx4\" (UniqueName: \"kubernetes.io/projected/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-kube-api-access-jqxx4\") pod \"payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67\" (UID: \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:04.078965 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:04.078910 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:04.404944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:04.404842 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:04.406307 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:04.406278 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:57:04.406470 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:04.406324 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 01:57:04.495113 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:04.495079 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67"] Oct 30 01:57:04.496927 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:57:04.496850 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b568f33_8f91_488d_9d8f_2232ef2bbf2c.slice/crio-d2328ed7f619a3892225c8fcdf4d3db26cc60f1348e4329f71caf9ae28ed44e2 WatchSource:0}: Error finding container d2328ed7f619a3892225c8fcdf4d3db26cc60f1348e4329f71caf9ae28ed44e2: Status 404 returned error can't find the container with id d2328ed7f619a3892225c8fcdf4d3db26cc60f1348e4329f71caf9ae28ed44e2 Oct 30 01:57:05.405276 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:05.405222 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:57:05.405490 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:05.405314 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 01:57:05.461742 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:05.461696 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" event={"ID":"3b568f33-8f91-488d-9d8f-2232ef2bbf2c","Type":"ContainerStarted","Data":"5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2"} Oct 30 01:57:05.461947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:05.461787 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" event={"ID":"3b568f33-8f91-488d-9d8f-2232ef2bbf2c","Type":"ContainerStarted","Data":"d2328ed7f619a3892225c8fcdf4d3db26cc60f1348e4329f71caf9ae28ed44e2"} Oct 30 01:57:05.461947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:05.461820 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:05.514106 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:05.514037 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" podStartSLOduration=2.514020853 podStartE2EDuration="2.514020853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:05.512506629 +0000 UTC m=+1704.015438852" watchObservedRunningTime="2026-10-30 01:57:05.514020853 +0000 UTC m=+1704.016953073" Oct 30 01:57:06.405695 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:06.405657 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:57:06.405956 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:06.405717 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 01:57:07.405252 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:07.405214 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:57:07.405835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:07.405275 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 01:57:08.405635 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:08.405597 5035 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:57:08.406127 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:08.405658 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 01:57:09.490603 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:09.490014 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:09.491129 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:09.491026 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:09.496549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:09.492001 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:11.994629 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:11.994581 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8"] Oct 30 01:57:12.000423 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.000384 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:12.029751 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.029699 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2933ff00-dockercfg-m78xx\"" Oct 30 01:57:12.030505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.030472 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2933ff00\"" Oct 30 01:57:12.035230 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.035190 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8"] Oct 30 01:57:12.181560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.145729 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12750093-5cf8-4f99-a8d4-1ed1caa6071c-plugins-config-volume\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8\" (UID: \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:12.181560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.145897 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dx7\" (UniqueName: \"kubernetes.io/projected/12750093-5cf8-4f99-a8d4-1ed1caa6071c-kube-api-access-24dx7\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8\" (UID: \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:12.213418 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.213375 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs"] Oct 30 01:57:12.232995 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.232958 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs"] Oct 30 01:57:12.235497 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.234479 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:12.262604 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.257852 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12750093-5cf8-4f99-a8d4-1ed1caa6071c-plugins-config-volume\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8\" (UID: \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:12.262604 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.258021 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dx7\" (UniqueName: \"kubernetes.io/projected/12750093-5cf8-4f99-a8d4-1ed1caa6071c-kube-api-access-24dx7\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8\" (UID: \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:12.262604 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.259343 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12750093-5cf8-4f99-a8d4-1ed1caa6071c-plugins-config-volume\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8\" (UID: \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:12.348949 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.348866 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dx7\" (UniqueName: \"kubernetes.io/projected/12750093-5cf8-4f99-a8d4-1ed1caa6071c-kube-api-access-24dx7\") pod \"payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8\" (UID: \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\") " pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:12.368582 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.367949 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4677ddd1-0240-4538-bfba-5d1122890c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs\" (UID: \"4677ddd1-0240-4538-bfba-5d1122890c56\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:12.369204 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.369134 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhf6x\" (UniqueName: \"kubernetes.io/projected/4677ddd1-0240-4538-bfba-5d1122890c56-kube-api-access-fhf6x\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs\" (UID: \"4677ddd1-0240-4538-bfba-5d1122890c56\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:12.471308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.471262 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhf6x\" (UniqueName: \"kubernetes.io/projected/4677ddd1-0240-4538-bfba-5d1122890c56-kube-api-access-fhf6x\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs\" (UID: \"4677ddd1-0240-4538-bfba-5d1122890c56\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:12.471495 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.471455 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4677ddd1-0240-4538-bfba-5d1122890c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs\" (UID: \"4677ddd1-0240-4538-bfba-5d1122890c56\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:12.472335 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.472302 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4677ddd1-0240-4538-bfba-5d1122890c56-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs\" (UID: \"4677ddd1-0240-4538-bfba-5d1122890c56\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:12.500404 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.500367 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhf6x\" (UniqueName: \"kubernetes.io/projected/4677ddd1-0240-4538-bfba-5d1122890c56-kube-api-access-fhf6x\") pod \"payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs\" (UID: \"4677ddd1-0240-4538-bfba-5d1122890c56\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:12.625646 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.620793 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:12.682385 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:12.682336 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:13.062806 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.062771 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8"] Oct 30 01:57:13.287042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.287011 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs"] Oct 30 01:57:13.287502 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:57:13.287480 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4677ddd1_0240_4538_bfba_5d1122890c56.slice/crio-c0408e66816b2d9d8245c6296c5cef9d8309b07ea64d51a90bcc716d1a6bdd15 WatchSource:0}: Error finding container c0408e66816b2d9d8245c6296c5cef9d8309b07ea64d51a90bcc716d1a6bdd15: Status 404 returned error can't find the container with id c0408e66816b2d9d8245c6296c5cef9d8309b07ea64d51a90bcc716d1a6bdd15 Oct 30 01:57:13.570484 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.570430 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" event={"ID":"12750093-5cf8-4f99-a8d4-1ed1caa6071c","Type":"ContainerStarted","Data":"607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f"} Oct 30 01:57:13.570694 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.570541 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" event={"ID":"12750093-5cf8-4f99-a8d4-1ed1caa6071c","Type":"ContainerStarted","Data":"090a2e519a8843150f4b748d308f9e6a6c8eb77f5388c33795c5f797fa389b58"} Oct 30 01:57:13.570694 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.570565 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:13.572422 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.572388 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" event={"ID":"4677ddd1-0240-4538-bfba-5d1122890c56","Type":"ContainerStarted","Data":"0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f"} Oct 30 01:57:13.572634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.572449 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" event={"ID":"4677ddd1-0240-4538-bfba-5d1122890c56","Type":"ContainerStarted","Data":"c0408e66816b2d9d8245c6296c5cef9d8309b07ea64d51a90bcc716d1a6bdd15"} Oct 30 01:57:13.572634 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.572483 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:13.629090 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.629016 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" podStartSLOduration=2.628999155 podStartE2EDuration="2.628999155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:13.628042502 +0000 UTC m=+1712.130974745" watchObservedRunningTime="2026-10-30 01:57:13.628999155 +0000 UTC m=+1712.131931375" Oct 30 01:57:13.680440 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:13.680365 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" podStartSLOduration=1.6803497680000001 podStartE2EDuration="1.680349768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:13.677763352 +0000 UTC m=+1712.180695572" watchObservedRunningTime="2026-10-30 01:57:13.680349768 +0000 UTC m=+1712.183281987" Oct 30 01:57:16.495286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:16.495249 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:57:17.654590 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:57:17.646781 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:57:21.053811 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.053765 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp"] Oct 30 01:57:21.058972 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.058944 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.089151 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.084177 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2818a6-serving-cert\"" Oct 30 01:57:21.104624 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.104583 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp"] Oct 30 01:57:21.128784 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.128740 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stzgv\" (UniqueName: \"kubernetes.io/projected/3f6b541e-f5e4-4332-9a41-bb57f4088585-kube-api-access-stzgv\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.129047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.128985 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-maas-api-tls\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.129101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.129056 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6b541e-f5e4-4332-9a41-bb57f4088585-openshift-service-ca\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.129101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.129073 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-metrics-certs\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.230765 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.230727 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stzgv\" (UniqueName: \"kubernetes.io/projected/3f6b541e-f5e4-4332-9a41-bb57f4088585-kube-api-access-stzgv\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.230963 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.230902 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-maas-api-tls\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.231057 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.230966 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6b541e-f5e4-4332-9a41-bb57f4088585-openshift-service-ca\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.231057 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.230983 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-metrics-certs\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.231805 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.231775 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6b541e-f5e4-4332-9a41-bb57f4088585-openshift-service-ca\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.233502 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.233471 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-maas-api-tls\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.233687 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.233643 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-metrics-certs\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.255896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.255855 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stzgv\" (UniqueName: \"kubernetes.io/projected/3f6b541e-f5e4-4332-9a41-bb57f4088585-kube-api-access-stzgv\") pod \"maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.392451 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.392330 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:21.727142 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:21.727109 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp"] Oct 30 01:57:21.727786 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:57:21.727758 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6b541e_f5e4_4332_9a41_bb57f4088585.slice/crio-2e641da1d1490dfc872bc5e3cfa917042743bb7c66b0e0154541f913d6f0706a WatchSource:0}: Error finding container 2e641da1d1490dfc872bc5e3cfa917042743bb7c66b0e0154541f913d6f0706a: Status 404 returned error can't find the container with id 2e641da1d1490dfc872bc5e3cfa917042743bb7c66b0e0154541f913d6f0706a Oct 30 01:57:22.648264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:22.648221 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" event={"ID":"3f6b541e-f5e4-4332-9a41-bb57f4088585","Type":"ContainerStarted","Data":"179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042"} Oct 30 01:57:22.648778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:22.648293 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" event={"ID":"3f6b541e-f5e4-4332-9a41-bb57f4088585","Type":"ContainerStarted","Data":"2e641da1d1490dfc872bc5e3cfa917042743bb7c66b0e0154541f913d6f0706a"} Oct 30 01:57:22.648778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:22.648322 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:22.699119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:22.699060 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" podStartSLOduration=1.321952191 podStartE2EDuration="1.699044085s" totalImagesPullingTime="377.091894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:22.694071023 +0000 UTC m=+1721.197003242" watchObservedRunningTime="2026-10-30 01:57:22.699044085 +0000 UTC m=+1721.201976305" Oct 30 01:57:24.583159 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:24.583127 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:57:24.583609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:24.583520 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:57:28.661560 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:28.661498 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:57:32.265586 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:32.265539 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn"] Oct 30 01:57:32.266190 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:32.265945 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerName="istio-proxy" containerID="cri-o://7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635" gracePeriod=30 Oct 30 01:57:32.279839 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:32.279800 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl"] Oct 30 01:57:32.280131 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:32.280095 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerName="istio-proxy" containerID="cri-o://a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540" gracePeriod=30 Oct 30 01:57:37.589663 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.589625 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:57:37.612896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.612856 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-credential-socket\") pod \"e37613cf-b692-4661-8206-a3385fd1c2f4\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " Oct 30 01:57:37.613047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.612912 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-token\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-token\") pod \"e37613cf-b692-4661-8206-a3385fd1c2f4\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " Oct 30 01:57:37.613047 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613039 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-podinfo\") pod \"e37613cf-b692-4661-8206-a3385fd1c2f4\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " Oct 30 01:57:37.613136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613113 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-envoy\") pod \"e37613cf-b692-4661-8206-a3385fd1c2f4\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " Oct 30 01:57:37.613178 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613153 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-data\") pod \"e37613cf-b692-4661-8206-a3385fd1c2f4\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " Oct 30 01:57:37.613258 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613147 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-credential-socket" pod "e37613cf-b692-4661-8206-a3385fd1c2f4" (UID: "e37613cf-b692-4661-8206-a3385fd1c2f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.613415 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613386 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-data" pod "e37613cf-b692-4661-8206-a3385fd1c2f4" (UID: "e37613cf-b692-4661-8206-a3385fd1c2f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.613522 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613400 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-certs\") pod \"e37613cf-b692-4661-8206-a3385fd1c2f4\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " Oct 30 01:57:37.613522 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613511 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-kube-api-access-dwmth\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-kube-api-access-dwmth\") pod \"e37613cf-b692-4661-8206-a3385fd1c2f4\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " Oct 30 01:57:37.613666 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613623 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-certs" pod "e37613cf-b692-4661-8206-a3385fd1c2f4" (UID: "e37613cf-b692-4661-8206-a3385fd1c2f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.613724 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613696 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37613cf-b692-4661-8206-a3385fd1c2f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37613cf-b692-4661-8206-a3385fd1c2f4-istiod-ca-cert\") pod \"e37613cf-b692-4661-8206-a3385fd1c2f4\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " Oct 30 01:57:37.613780 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.613735 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-socket\") pod \"e37613cf-b692-4661-8206-a3385fd1c2f4\" (UID: \"e37613cf-b692-4661-8206-a3385fd1c2f4\") " Oct 30 01:57:37.614264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.614225 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37613cf-b692-4661-8206-a3385fd1c2f4-istiod-ca-cert" pod "e37613cf-b692-4661-8206-a3385fd1c2f4" (UID: "e37613cf-b692-4661-8206-a3385fd1c2f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:37.614410 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.614265 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-socket" pod "e37613cf-b692-4661-8206-a3385fd1c2f4" (UID: "e37613cf-b692-4661-8206-a3385fd1c2f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.614799 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.614774 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.614899 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.614804 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37613cf-b692-4661-8206-a3385fd1c2f4-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.614899 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.614821 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.614899 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.614838 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.614899 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.614855 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.615358 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.615333 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-token" pod "e37613cf-b692-4661-8206-a3385fd1c2f4" (UID: "e37613cf-b692-4661-8206-a3385fd1c2f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.615507 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.615469 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-envoy" pod "e37613cf-b692-4661-8206-a3385fd1c2f4" (UID: "e37613cf-b692-4661-8206-a3385fd1c2f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.615857 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.615833 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-kube-api-access-dwmth" pod "e37613cf-b692-4661-8206-a3385fd1c2f4" (UID: "e37613cf-b692-4661-8206-a3385fd1c2f4"). InnerVolumeSpecName "kube-api-access-dwmth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.615968 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.615926 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-podinfo" pod "e37613cf-b692-4661-8206-a3385fd1c2f4" (UID: "e37613cf-b692-4661-8206-a3385fd1c2f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:57:37.651819 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.651784 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:37.715591 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.715550 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27114de7-baaf-4f00-aef5-146ea1b7d580-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27114de7-baaf-4f00-aef5-146ea1b7d580-istiod-ca-cert\") pod \"27114de7-baaf-4f00-aef5-146ea1b7d580\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " Oct 30 01:57:37.715810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.715633 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-certs\") pod \"27114de7-baaf-4f00-aef5-146ea1b7d580\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " Oct 30 01:57:37.715810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.715688 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-socket\") pod \"27114de7-baaf-4f00-aef5-146ea1b7d580\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " Oct 30 01:57:37.715810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.715730 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-envoy\") pod \"27114de7-baaf-4f00-aef5-146ea1b7d580\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " Oct 30 01:57:37.715810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.715757 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-token\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-token\") pod \"27114de7-baaf-4f00-aef5-146ea1b7d580\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " Oct 30 01:57:37.715810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.715802 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-credential-socket\") pod \"27114de7-baaf-4f00-aef5-146ea1b7d580\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " Oct 30 01:57:37.716119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.715870 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-data\") pod \"27114de7-baaf-4f00-aef5-146ea1b7d580\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " Oct 30 01:57:37.716119 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.715923 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-kube-api-access-bdf5h\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-kube-api-access-bdf5h\") pod \"27114de7-baaf-4f00-aef5-146ea1b7d580\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " Oct 30 01:57:37.716228 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.716127 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-podinfo\") pod \"27114de7-baaf-4f00-aef5-146ea1b7d580\" (UID: \"27114de7-baaf-4f00-aef5-146ea1b7d580\") " Oct 30 01:57:37.716378 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.716349 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-socket" pod "27114de7-baaf-4f00-aef5-146ea1b7d580" (UID: "27114de7-baaf-4f00-aef5-146ea1b7d580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.716512 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.716486 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-data" pod "27114de7-baaf-4f00-aef5-146ea1b7d580" (UID: "27114de7-baaf-4f00-aef5-146ea1b7d580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.717153 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.717126 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmth\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-kube-api-access-dwmth\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.717308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.717158 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.717308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.717173 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.717308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.717189 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.717308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.717203 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37613cf-b692-4661-8206-a3385fd1c2f4-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.717308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.717218 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.717308 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.717257 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-credential-socket" pod "27114de7-baaf-4f00-aef5-146ea1b7d580" (UID: "27114de7-baaf-4f00-aef5-146ea1b7d580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.722947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.718246 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-certs" pod "27114de7-baaf-4f00-aef5-146ea1b7d580" (UID: "27114de7-baaf-4f00-aef5-146ea1b7d580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.722947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.718270 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27114de7-baaf-4f00-aef5-146ea1b7d580-istiod-ca-cert" pod "27114de7-baaf-4f00-aef5-146ea1b7d580" (UID: "27114de7-baaf-4f00-aef5-146ea1b7d580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:37.722947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.720206 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-token" pod "27114de7-baaf-4f00-aef5-146ea1b7d580" (UID: "27114de7-baaf-4f00-aef5-146ea1b7d580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.722947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.720252 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-kube-api-access-bdf5h" pod "27114de7-baaf-4f00-aef5-146ea1b7d580" (UID: "27114de7-baaf-4f00-aef5-146ea1b7d580"). InnerVolumeSpecName "kube-api-access-bdf5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.722947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.720700 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-envoy" pod "27114de7-baaf-4f00-aef5-146ea1b7d580" (UID: "27114de7-baaf-4f00-aef5-146ea1b7d580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.722947 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.721838 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-podinfo" pod "27114de7-baaf-4f00-aef5-146ea1b7d580" (UID: "27114de7-baaf-4f00-aef5-146ea1b7d580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:57:37.819448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.819354 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.819448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.819392 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27114de7-baaf-4f00-aef5-146ea1b7d580-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.819448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.819403 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.819448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.819411 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.819448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.819420 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.819448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.819428 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27114de7-baaf-4f00-aef5-146ea1b7d580-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.819448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.819438 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdf5h\" (UniqueName: \"kubernetes.io/projected/27114de7-baaf-4f00-aef5-146ea1b7d580-kube-api-access-bdf5h\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.872984 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.872942 5035 generic.go:356] "Generic (PLEG): container finished" podID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerID="7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635" exitCode=0 Oct 30 01:57:37.873166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.872996 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" Oct 30 01:57:37.873166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.873016 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" event={"ID":"e37613cf-b692-4661-8206-a3385fd1c2f4","Type":"ContainerDied","Data":"7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635"} Oct 30 01:57:37.873166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.873073 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn" event={"ID":"e37613cf-b692-4661-8206-a3385fd1c2f4","Type":"ContainerDied","Data":"e53258ae75b897b87776040ef8f7ab4e2fb186315ed8d5a865cca6a591d0d296"} Oct 30 01:57:37.873166 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.873100 5035 scope.go:122] "RemoveContainer" containerID="7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635" Oct 30 01:57:37.874953 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.874926 5035 generic.go:356] "Generic (PLEG): container finished" podID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerID="a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540" exitCode=0 Oct 30 01:57:37.875100 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.874967 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" Oct 30 01:57:37.875100 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.875014 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" event={"ID":"27114de7-baaf-4f00-aef5-146ea1b7d580","Type":"ContainerDied","Data":"a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540"} Oct 30 01:57:37.875100 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.875061 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl" event={"ID":"27114de7-baaf-4f00-aef5-146ea1b7d580","Type":"ContainerDied","Data":"55d503a766186b4b70c6239431ec4a6c6e79c0bc05607a0ad0b959f255775f71"} Oct 30 01:57:37.885945 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.885923 5035 scope.go:122] "RemoveContainer" containerID="7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635" Oct 30 01:57:37.886313 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:57:37.886291 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635\": container with ID starting with 7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635 not found: ID does not exist" containerID="7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635" Oct 30 01:57:37.886418 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.886322 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635"} err="failed to get container status \"7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635\": rpc error: code = NotFound desc = could not find container \"7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635\": container with ID starting with 7c30a031ff2e431993c7598923691767f28913830c459a6210def3274855e635 not found: ID does not exist" Oct 30 01:57:37.886418 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.886344 5035 scope.go:122] "RemoveContainer" containerID="a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540" Oct 30 01:57:37.903477 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.903451 5035 scope.go:122] "RemoveContainer" containerID="a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540" Oct 30 01:57:37.903901 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:57:37.903873 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540\": container with ID starting with a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540 not found: ID does not exist" containerID="a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540" Oct 30 01:57:37.903997 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.903919 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540"} err="failed to get container status \"a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540\": rpc error: code = NotFound desc = could not find container \"a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540\": container with ID starting with a69a2c8df10e994b499a8ba80d49525256e86133af52c9ab5618871971e17540 not found: ID does not exist" Oct 30 01:57:37.933287 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.933249 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn"] Oct 30 01:57:37.937472 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.937439 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-qftpn"] Oct 30 01:57:37.978960 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.978921 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl"] Oct 30 01:57:37.983408 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:37.983373 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-zd5wl"] Oct 30 01:57:38.782323 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:38.782277 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr"] Oct 30 01:57:39.647302 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:39.647267 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" path="/var/lib/kubelet/pods/27114de7-baaf-4f00-aef5-146ea1b7d580/volumes" Oct 30 01:57:39.647954 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:39.647930 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" path="/var/lib/kubelet/pods/e37613cf-b692-4661-8206-a3385fd1c2f4/volumes" Oct 30 01:57:42.511967 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:42.511887 5035 scope.go:122] "RemoveContainer" containerID="b5c681775b4bc02050f7e4765105a3cb2bc4c93ac32a277b2825ab8597fe4c4c" Oct 30 01:57:42.531744 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:42.531714 5035 scope.go:122] "RemoveContainer" containerID="78657004f70f4de664934c158b96a1a1851ba5baf20231472613f5986090cc43" Oct 30 01:57:42.542354 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:42.542321 5035 scope.go:122] "RemoveContainer" containerID="44e1bc1c48f502c932bb85f1e9be7ccb2c1f334b32ad21f21074d4677af8ae4d" Oct 30 01:57:43.783309 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:43.783238 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" podUID="c5f89f71-2cbe-4483-a4e0-78723415f1f9" containerName="payload-processing" containerID="cri-o://e3f94ec15a68b44e47ff4275604450beaf100c8bf07d0bad58b992b6ea45a92b" gracePeriod=25 Oct 30 01:57:44.006187 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:43.998935 5035 generic.go:356] "Generic (PLEG): container finished" podID="c5f89f71-2cbe-4483-a4e0-78723415f1f9" containerID="e3f94ec15a68b44e47ff4275604450beaf100c8bf07d0bad58b992b6ea45a92b" exitCode=0 Oct 30 01:57:44.006187 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:43.999114 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" event={"ID":"c5f89f71-2cbe-4483-a4e0-78723415f1f9","Type":"ContainerDied","Data":"e3f94ec15a68b44e47ff4275604450beaf100c8bf07d0bad58b992b6ea45a92b"} Oct 30 01:57:44.215246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:44.215213 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:57:44.354816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:44.354719 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f89f71-2cbe-4483-a4e0-78723415f1f9-kube-api-access-qxxfp\" (UniqueName: \"kubernetes.io/projected/c5f89f71-2cbe-4483-a4e0-78723415f1f9-kube-api-access-qxxfp\") pod \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\" (UID: \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\") " Oct 30 01:57:44.354816 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:44.354787 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f89f71-2cbe-4483-a4e0-78723415f1f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f89f71-2cbe-4483-a4e0-78723415f1f9-plugins-config-volume\") pod \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\" (UID: \"c5f89f71-2cbe-4483-a4e0-78723415f1f9\") " Oct 30 01:57:44.355226 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:44.355200 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f89f71-2cbe-4483-a4e0-78723415f1f9-plugins-config-volume" pod "c5f89f71-2cbe-4483-a4e0-78723415f1f9" (UID: "c5f89f71-2cbe-4483-a4e0-78723415f1f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:44.355727 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:44.355701 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f89f71-2cbe-4483-a4e0-78723415f1f9-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:44.357203 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:44.357177 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f89f71-2cbe-4483-a4e0-78723415f1f9-kube-api-access-qxxfp" pod "c5f89f71-2cbe-4483-a4e0-78723415f1f9" (UID: "c5f89f71-2cbe-4483-a4e0-78723415f1f9"). InnerVolumeSpecName "kube-api-access-qxxfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:44.458039 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:44.458000 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxxfp\" (UniqueName: \"kubernetes.io/projected/c5f89f71-2cbe-4483-a4e0-78723415f1f9-kube-api-access-qxxfp\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:57:45.015219 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:45.015160 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" event={"ID":"c5f89f71-2cbe-4483-a4e0-78723415f1f9","Type":"ContainerDied","Data":"ea6769fc92deb5fb6558b4627277e5c6ed299cc2a8fb0eacd21ffd78b29e7162"} Oct 30 01:57:45.015720 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:45.015415 5035 scope.go:122] "RemoveContainer" containerID="e3f94ec15a68b44e47ff4275604450beaf100c8bf07d0bad58b992b6ea45a92b" Oct 30 01:57:45.015720 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:45.015444 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr" Oct 30 01:57:45.072432 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:45.072395 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr"] Oct 30 01:57:45.085800 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:45.085756 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b2c11f-777dbfbc54-n2jtr"] Oct 30 01:57:45.644105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:57:45.644070 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f89f71-2cbe-4483-a4e0-78723415f1f9" path="/var/lib/kubelet/pods/c5f89f71-2cbe-4483-a4e0-78723415f1f9/volumes" Oct 30 01:57:47.651091 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:57:47.650196 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:58:10.047311 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:10.047272 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr"] Oct 30 01:58:10.059603 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:10.059570 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82948b86-bsrwr"] Oct 30 01:58:11.645406 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:11.645369 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd8ad70-8874-49aa-970f-7f58401c2efe" path="/var/lib/kubelet/pods/7bd8ad70-8874-49aa-970f-7f58401c2efe/volumes" Oct 30 01:58:17.631262 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:58:17.631221 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:58:19.185598 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:19.185557 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67"] Oct 30 01:58:23.041758 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:23.041717 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws"] Oct 30 01:58:23.055276 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:23.055240 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df25ae-fj8ws"] Oct 30 01:58:23.645027 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:23.644988 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc882188-78c5-4f3b-b982-167506d515cc" path="/var/lib/kubelet/pods/dc882188-78c5-4f3b-b982-167506d515cc/volumes" Oct 30 01:58:24.359271 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:24.359224 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" podUID="3b568f33-8f91-488d-9d8f-2232ef2bbf2c" containerName="payload-pre-processing" containerID="cri-o://5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2" gracePeriod=25 Oct 30 01:58:24.691063 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:24.691031 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:58:24.799788 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:24.799748 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-plugins-config-volume\") pod \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\" (UID: \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\") " Oct 30 01:58:24.800049 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:24.800029 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-kube-api-access-jqxx4\" (UniqueName: \"kubernetes.io/projected/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-kube-api-access-jqxx4\") pod \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\" (UID: \"3b568f33-8f91-488d-9d8f-2232ef2bbf2c\") " Oct 30 01:58:24.800154 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:24.800131 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-plugins-config-volume" pod "3b568f33-8f91-488d-9d8f-2232ef2bbf2c" (UID: "3b568f33-8f91-488d-9d8f-2232ef2bbf2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:24.800665 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:24.800645 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:24.802483 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:24.802456 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-kube-api-access-jqxx4" pod "3b568f33-8f91-488d-9d8f-2232ef2bbf2c" (UID: "3b568f33-8f91-488d-9d8f-2232ef2bbf2c"). InnerVolumeSpecName "kube-api-access-jqxx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:24.902541 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:24.902494 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqxx4\" (UniqueName: \"kubernetes.io/projected/3b568f33-8f91-488d-9d8f-2232ef2bbf2c-kube-api-access-jqxx4\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:25.405032 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.404994 5035 generic.go:356] "Generic (PLEG): container finished" podID="3b568f33-8f91-488d-9d8f-2232ef2bbf2c" containerID="5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2" exitCode=0 Oct 30 01:58:25.405032 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.405037 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" event={"ID":"3b568f33-8f91-488d-9d8f-2232ef2bbf2c","Type":"ContainerDied","Data":"5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2"} Oct 30 01:58:25.405620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.405047 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" Oct 30 01:58:25.405620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.405082 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67" event={"ID":"3b568f33-8f91-488d-9d8f-2232ef2bbf2c","Type":"ContainerDied","Data":"d2328ed7f619a3892225c8fcdf4d3db26cc60f1348e4329f71caf9ae28ed44e2"} Oct 30 01:58:25.405620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.405110 5035 scope.go:122] "RemoveContainer" containerID="5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2" Oct 30 01:58:25.416248 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.416224 5035 scope.go:122] "RemoveContainer" containerID="5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2" Oct 30 01:58:25.416591 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:58:25.416565 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2\": container with ID starting with 5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2 not found: ID does not exist" containerID="5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2" Oct 30 01:58:25.416658 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.416601 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2"} err="failed to get container status \"5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2\": rpc error: code = NotFound desc = could not find container \"5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2\": container with ID starting with 5fb27fe7169e0757190aeaa96e930663f29e3a37a8ad24e160311fd538c554a2 not found: ID does not exist" Oct 30 01:58:25.461830 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.461767 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67"] Oct 30 01:58:25.474819 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.474774 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2818a6-586fb4d987-bvv67"] Oct 30 01:58:25.644778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:25.644740 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b568f33-8f91-488d-9d8f-2232ef2bbf2c" path="/var/lib/kubelet/pods/3b568f33-8f91-488d-9d8f-2232ef2bbf2c/volumes" Oct 30 01:58:36.768571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:36.766162 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp"] Oct 30 01:58:36.768571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:36.766452 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" podUID="3f6b541e-f5e4-4332-9a41-bb57f4088585" containerName="maas-api" containerID="cri-o://179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042" gracePeriod=30 Oct 30 01:58:37.056101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.056002 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m"] Oct 30 01:58:37.060555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.060507 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b05c63-9zc6m"] Oct 30 01:58:37.101914 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.101880 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:58:37.205943 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.205903 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw"] Oct 30 01:58:37.207159 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207128 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c5f89f71-2cbe-4483-a4e0-78723415f1f9" containerName="payload-processing" Oct 30 01:58:37.207159 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207160 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f89f71-2cbe-4483-a4e0-78723415f1f9" containerName="payload-processing" Oct 30 01:58:37.207323 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207209 5035 cpu_manager.go:414] "containerMap: removing container" podUID="3b568f33-8f91-488d-9d8f-2232ef2bbf2c" containerName="payload-pre-processing" Oct 30 01:58:37.207323 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207218 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b568f33-8f91-488d-9d8f-2232ef2bbf2c" containerName="payload-pre-processing" Oct 30 01:58:37.207323 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207291 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerName="istio-proxy" Oct 30 01:58:37.207323 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207300 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerName="istio-proxy" Oct 30 01:58:37.207562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207344 5035 cpu_manager.go:414] "containerMap: removing container" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerName="istio-proxy" Oct 30 01:58:37.207562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207352 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerName="istio-proxy" Oct 30 01:58:37.207562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207377 5035 cpu_manager.go:414] "containerMap: removing container" podUID="3f6b541e-f5e4-4332-9a41-bb57f4088585" containerName="maas-api" Oct 30 01:58:37.207562 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207386 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6b541e-f5e4-4332-9a41-bb57f4088585" containerName="maas-api" Oct 30 01:58:37.207699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207598 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b568f33-8f91-488d-9d8f-2232ef2bbf2c" containerName="payload-pre-processing" Oct 30 01:58:37.207699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207622 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37613cf-b692-4661-8206-a3385fd1c2f4" containerName="istio-proxy" Oct 30 01:58:37.207699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207637 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="27114de7-baaf-4f00-aef5-146ea1b7d580" containerName="istio-proxy" Oct 30 01:58:37.207699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207647 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6b541e-f5e4-4332-9a41-bb57f4088585" containerName="maas-api" Oct 30 01:58:37.207699 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.207663 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f89f71-2cbe-4483-a4e0-78723415f1f9" containerName="payload-processing" Oct 30 01:58:37.213244 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.212707 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:37.233078 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.232848 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw"] Oct 30 01:58:37.233078 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.233028 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86dd79\"" Oct 30 01:58:37.233296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.233118 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86dd79-dockercfg-cxn4v\"" Oct 30 01:58:37.254312 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.254275 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-maas-api-tls\") pod \"3f6b541e-f5e4-4332-9a41-bb57f4088585\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " Oct 30 01:58:37.254518 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.254369 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-metrics-certs\") pod \"3f6b541e-f5e4-4332-9a41-bb57f4088585\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " Oct 30 01:58:37.255001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.254598 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6b541e-f5e4-4332-9a41-bb57f4088585-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6b541e-f5e4-4332-9a41-bb57f4088585-openshift-service-ca\") pod \"3f6b541e-f5e4-4332-9a41-bb57f4088585\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " Oct 30 01:58:37.255001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.254678 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6b541e-f5e4-4332-9a41-bb57f4088585-kube-api-access-stzgv\" (UniqueName: \"kubernetes.io/projected/3f6b541e-f5e4-4332-9a41-bb57f4088585-kube-api-access-stzgv\") pod \"3f6b541e-f5e4-4332-9a41-bb57f4088585\" (UID: \"3f6b541e-f5e4-4332-9a41-bb57f4088585\") " Oct 30 01:58:37.255001 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.254948 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6b541e-f5e4-4332-9a41-bb57f4088585-openshift-service-ca" pod "3f6b541e-f5e4-4332-9a41-bb57f4088585" (UID: "3f6b541e-f5e4-4332-9a41-bb57f4088585"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:37.255586 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.255560 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6b541e-f5e4-4332-9a41-bb57f4088585-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:37.259348 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.258846 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-maas-api-tls" pod "3f6b541e-f5e4-4332-9a41-bb57f4088585" (UID: "3f6b541e-f5e4-4332-9a41-bb57f4088585"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:58:37.259348 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.259306 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-metrics-certs" pod "3f6b541e-f5e4-4332-9a41-bb57f4088585" (UID: "3f6b541e-f5e4-4332-9a41-bb57f4088585"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:58:37.260979 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.260938 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6b541e-f5e4-4332-9a41-bb57f4088585-kube-api-access-stzgv" pod "3f6b541e-f5e4-4332-9a41-bb57f4088585" (UID: "3f6b541e-f5e4-4332-9a41-bb57f4088585"). InnerVolumeSpecName "kube-api-access-stzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:37.405049 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.402947 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d469ef37-96c3-434b-9d00-7a8771832df7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw\" (UID: \"d469ef37-96c3-434b-9d00-7a8771832df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:37.405615 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.405457 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktmn\" (UniqueName: \"kubernetes.io/projected/d469ef37-96c3-434b-9d00-7a8771832df7-kube-api-access-hktmn\") pod \"payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw\" (UID: \"d469ef37-96c3-434b-9d00-7a8771832df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:37.406506 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.406373 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:37.406688 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.406671 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f6b541e-f5e4-4332-9a41-bb57f4088585-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:37.406778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.406766 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stzgv\" (UniqueName: \"kubernetes.io/projected/3f6b541e-f5e4-4332-9a41-bb57f4088585-kube-api-access-stzgv\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:37.510411 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.510365 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d469ef37-96c3-434b-9d00-7a8771832df7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw\" (UID: \"d469ef37-96c3-434b-9d00-7a8771832df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:37.510736 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.510713 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktmn\" (UniqueName: \"kubernetes.io/projected/d469ef37-96c3-434b-9d00-7a8771832df7-kube-api-access-hktmn\") pod \"payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw\" (UID: \"d469ef37-96c3-434b-9d00-7a8771832df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:37.512158 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.512099 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d469ef37-96c3-434b-9d00-7a8771832df7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw\" (UID: \"d469ef37-96c3-434b-9d00-7a8771832df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:37.536215 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.536178 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktmn\" (UniqueName: \"kubernetes.io/projected/d469ef37-96c3-434b-9d00-7a8771832df7-kube-api-access-hktmn\") pod \"payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw\" (UID: \"d469ef37-96c3-434b-9d00-7a8771832df7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:37.543798 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.543760 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:37.589263 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.589226 5035 generic.go:356] "Generic (PLEG): container finished" podID="3f6b541e-f5e4-4332-9a41-bb57f4088585" containerID="179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042" exitCode=0 Oct 30 01:58:37.589439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.589277 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" Oct 30 01:58:37.589439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.589312 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" event={"ID":"3f6b541e-f5e4-4332-9a41-bb57f4088585","Type":"ContainerDied","Data":"179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042"} Oct 30 01:58:37.589439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.589375 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp" event={"ID":"3f6b541e-f5e4-4332-9a41-bb57f4088585","Type":"ContainerDied","Data":"2e641da1d1490dfc872bc5e3cfa917042743bb7c66b0e0154541f913d6f0706a"} Oct 30 01:58:37.589439 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.589399 5035 scope.go:122] "RemoveContainer" containerID="179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042" Oct 30 01:58:37.604518 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.604490 5035 scope.go:122] "RemoveContainer" containerID="179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042" Oct 30 01:58:37.604885 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:58:37.604861 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042\": container with ID starting with 179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042 not found: ID does not exist" containerID="179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042" Oct 30 01:58:37.604987 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.604896 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042"} err="failed to get container status \"179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042\": rpc error: code = NotFound desc = could not find container \"179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042\": container with ID starting with 179ba6d1199734b3c141d3b012c490dae664592e8b764e0e4a13a6cf77aea042 not found: ID does not exist" Oct 30 01:58:37.652901 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.652862 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d6ac09-9d47-4086-85a5-56a1fa7c101d" path="/var/lib/kubelet/pods/61d6ac09-9d47-4086-85a5-56a1fa7c101d/volumes" Oct 30 01:58:37.688762 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.688664 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp"] Oct 30 01:58:37.699775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.699733 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2818a6-5df8d5949c-rsvlp"] Oct 30 01:58:37.753177 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.753135 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8"] Oct 30 01:58:37.793413 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.793365 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs"] Oct 30 01:58:37.870740 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:37.870696 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw"] Oct 30 01:58:38.626246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:38.626203 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" event={"ID":"d469ef37-96c3-434b-9d00-7a8771832df7","Type":"ContainerStarted","Data":"8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d"} Oct 30 01:58:38.626485 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:38.626293 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" event={"ID":"d469ef37-96c3-434b-9d00-7a8771832df7","Type":"ContainerStarted","Data":"26fcf9e2c68366fd77b7acab7e7dd0161e59bd95240b48017e9ce9918c9db65c"} Oct 30 01:58:38.626593 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:38.626492 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:38.676822 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:38.676752 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" podStartSLOduration=1.676727723 podStartE2EDuration="1.676727723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:38.671759675 +0000 UTC m=+1797.174691912" watchObservedRunningTime="2026-10-30 01:58:38.676727723 +0000 UTC m=+1797.179659947" Oct 30 01:58:39.643857 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:39.643810 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6b541e-f5e4-4332-9a41-bb57f4088585" path="/var/lib/kubelet/pods/3f6b541e-f5e4-4332-9a41-bb57f4088585/volumes" Oct 30 01:58:42.754540 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:42.754481 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" podUID="12750093-5cf8-4f99-a8d4-1ed1caa6071c" containerName="payload-processing" containerID="cri-o://607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f" gracePeriod=25 Oct 30 01:58:42.793840 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:42.793796 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" podUID="4677ddd1-0240-4538-bfba-5d1122890c56" containerName="payload-pre-processing" containerID="cri-o://0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f" gracePeriod=25 Oct 30 01:58:43.117980 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.117947 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:58:43.174080 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.174052 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:58:43.330551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.327900 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12750093-5cf8-4f99-a8d4-1ed1caa6071c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12750093-5cf8-4f99-a8d4-1ed1caa6071c-plugins-config-volume\") pod \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\" (UID: \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\") " Oct 30 01:58:43.330551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.328219 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4677ddd1-0240-4538-bfba-5d1122890c56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4677ddd1-0240-4538-bfba-5d1122890c56-plugins-config-volume\") pod \"4677ddd1-0240-4538-bfba-5d1122890c56\" (UID: \"4677ddd1-0240-4538-bfba-5d1122890c56\") " Oct 30 01:58:43.330551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.328268 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12750093-5cf8-4f99-a8d4-1ed1caa6071c-kube-api-access-24dx7\" (UniqueName: \"kubernetes.io/projected/12750093-5cf8-4f99-a8d4-1ed1caa6071c-kube-api-access-24dx7\") pod \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\" (UID: \"12750093-5cf8-4f99-a8d4-1ed1caa6071c\") " Oct 30 01:58:43.330551 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.328789 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4677ddd1-0240-4538-bfba-5d1122890c56-kube-api-access-fhf6x\" (UniqueName: \"kubernetes.io/projected/4677ddd1-0240-4538-bfba-5d1122890c56-kube-api-access-fhf6x\") pod \"4677ddd1-0240-4538-bfba-5d1122890c56\" (UID: \"4677ddd1-0240-4538-bfba-5d1122890c56\") " Oct 30 01:58:43.339704 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.337911 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12750093-5cf8-4f99-a8d4-1ed1caa6071c-plugins-config-volume" pod "12750093-5cf8-4f99-a8d4-1ed1caa6071c" (UID: "12750093-5cf8-4f99-a8d4-1ed1caa6071c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:43.340195 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.340118 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4677ddd1-0240-4538-bfba-5d1122890c56-plugins-config-volume" pod "4677ddd1-0240-4538-bfba-5d1122890c56" (UID: "4677ddd1-0240-4538-bfba-5d1122890c56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:43.364805 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.364763 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4677ddd1-0240-4538-bfba-5d1122890c56-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:43.374842 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.374807 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12750093-5cf8-4f99-a8d4-1ed1caa6071c-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:43.395656 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.393963 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12750093-5cf8-4f99-a8d4-1ed1caa6071c-kube-api-access-24dx7" pod "12750093-5cf8-4f99-a8d4-1ed1caa6071c" (UID: "12750093-5cf8-4f99-a8d4-1ed1caa6071c"). InnerVolumeSpecName "kube-api-access-24dx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:43.426426 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.426330 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4677ddd1-0240-4538-bfba-5d1122890c56-kube-api-access-fhf6x" pod "4677ddd1-0240-4538-bfba-5d1122890c56" (UID: "4677ddd1-0240-4538-bfba-5d1122890c56"). InnerVolumeSpecName "kube-api-access-fhf6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:43.477015 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.476974 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dx7\" (UniqueName: \"kubernetes.io/projected/12750093-5cf8-4f99-a8d4-1ed1caa6071c-kube-api-access-24dx7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:43.477015 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.477011 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhf6x\" (UniqueName: \"kubernetes.io/projected/4677ddd1-0240-4538-bfba-5d1122890c56-kube-api-access-fhf6x\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:58:43.608510 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.608468 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 01:58:43.636159 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.635423 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:58:43.641787 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.641756 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 01:58:43.663637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.663613 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 01:58:43.666220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.666183 5035 generic.go:356] "Generic (PLEG): container finished" podID="12750093-5cf8-4f99-a8d4-1ed1caa6071c" containerID="607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f" exitCode=0 Oct 30 01:58:43.666390 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.666220 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" event={"ID":"12750093-5cf8-4f99-a8d4-1ed1caa6071c","Type":"ContainerDied","Data":"607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f"} Oct 30 01:58:43.666390 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.666250 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" Oct 30 01:58:43.666390 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.666290 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8" event={"ID":"12750093-5cf8-4f99-a8d4-1ed1caa6071c","Type":"ContainerDied","Data":"090a2e519a8843150f4b748d308f9e6a6c8eb77f5388c33795c5f797fa389b58"} Oct 30 01:58:43.666390 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.666323 5035 scope.go:122] "RemoveContainer" containerID="607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f" Oct 30 01:58:43.668155 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.667879 5035 generic.go:356] "Generic (PLEG): container finished" podID="4677ddd1-0240-4538-bfba-5d1122890c56" containerID="0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f" exitCode=0 Oct 30 01:58:43.668155 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.667971 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" Oct 30 01:58:43.668155 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.667966 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" event={"ID":"4677ddd1-0240-4538-bfba-5d1122890c56","Type":"ContainerDied","Data":"0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f"} Oct 30 01:58:43.668155 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.668103 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs" event={"ID":"4677ddd1-0240-4538-bfba-5d1122890c56","Type":"ContainerDied","Data":"c0408e66816b2d9d8245c6296c5cef9d8309b07ea64d51a90bcc716d1a6bdd15"} Oct 30 01:58:43.679334 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.679297 5035 scope.go:122] "RemoveContainer" containerID="607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f" Oct 30 01:58:43.679628 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:58:43.679601 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f\": container with ID starting with 607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f not found: ID does not exist" containerID="607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f" Oct 30 01:58:43.679718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.679636 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f"} err="failed to get container status \"607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f\": rpc error: code = NotFound desc = could not find container \"607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f\": container with ID starting with 607d9726316a4cfe2261699e838fddd986f431db64c05c57f311056e1213994f not found: ID does not exist" Oct 30 01:58:43.679718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.679659 5035 scope.go:122] "RemoveContainer" containerID="0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f" Oct 30 01:58:43.690369 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.690343 5035 scope.go:122] "RemoveContainer" containerID="0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f" Oct 30 01:58:43.690753 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:58:43.690728 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f\": container with ID starting with 0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f not found: ID does not exist" containerID="0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f" Oct 30 01:58:43.690846 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.690765 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f"} err="failed to get container status \"0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f\": rpc error: code = NotFound desc = could not find container \"0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f\": container with ID starting with 0ff014c1553df9b6880d7cc089e87aa1ba0c8f018fe3895bdf012e4e6a89345f not found: ID does not exist" Oct 30 01:58:43.723029 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.722990 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8"] Oct 30 01:58:43.735951 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.735916 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2933ff00-7574c49c54-nnhq8"] Oct 30 01:58:43.763496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.763387 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs"] Oct 30 01:58:43.768891 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:43.768858 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2933ff00-66f5ddbb99-v52zs"] Oct 30 01:58:45.645776 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:45.645736 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12750093-5cf8-4f99-a8d4-1ed1caa6071c" path="/var/lib/kubelet/pods/12750093-5cf8-4f99-a8d4-1ed1caa6071c/volumes" Oct 30 01:58:45.646433 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:45.646410 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4677ddd1-0240-4538-bfba-5d1122890c56" path="/var/lib/kubelet/pods/4677ddd1-0240-4538-bfba-5d1122890c56/volumes" Oct 30 01:58:47.636725 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:58:47.636680 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:58:49.645903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:49.645865 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:58:53.047222 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:53.047169 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp"] Oct 30 01:58:53.052837 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:53.052788 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2dfaee-gcbdp"] Oct 30 01:58:53.645336 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:53.645300 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd698533-6fa7-4af0-9775-ea05d2a896e4" path="/var/lib/kubelet/pods/fd698533-6fa7-4af0-9775-ea05d2a896e4/volumes" Oct 30 01:58:56.077344 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.077310 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx"] Oct 30 01:58:56.077999 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.077984 5035 cpu_manager.go:414] "containerMap: removing container" podUID="4677ddd1-0240-4538-bfba-5d1122890c56" containerName="payload-pre-processing" Oct 30 01:58:56.078040 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.078000 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4677ddd1-0240-4538-bfba-5d1122890c56" containerName="payload-pre-processing" Oct 30 01:58:56.078076 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.078040 5035 cpu_manager.go:414] "containerMap: removing container" podUID="12750093-5cf8-4f99-a8d4-1ed1caa6071c" containerName="payload-processing" Oct 30 01:58:56.078076 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.078045 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12750093-5cf8-4f99-a8d4-1ed1caa6071c" containerName="payload-processing" Oct 30 01:58:56.078154 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.078144 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="4677ddd1-0240-4538-bfba-5d1122890c56" containerName="payload-pre-processing" Oct 30 01:58:56.078185 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.078161 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="12750093-5cf8-4f99-a8d4-1ed1caa6071c" containerName="payload-processing" Oct 30 01:58:56.083609 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.083585 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:58:56.100418 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.100386 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-954773\"" Oct 30 01:58:56.100920 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.100891 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-954773-dockercfg-6d5bs\"" Oct 30 01:58:56.103319 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.103293 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx"] Oct 30 01:58:56.118211 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.118167 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm"] Oct 30 01:58:56.124010 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.123978 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:58:56.140496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.140454 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm"] Oct 30 01:58:56.149891 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.149854 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0120d-39b7-49e7-afe8-658da51a9a0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx\" (UID: \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:58:56.150087 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.150058 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8wt\" (UniqueName: \"kubernetes.io/projected/c1b0120d-39b7-49e7-afe8-658da51a9a0c-kube-api-access-7c8wt\") pod \"payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx\" (UID: \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:58:56.276759 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.276599 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0120d-39b7-49e7-afe8-658da51a9a0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx\" (UID: \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:58:56.277900 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.277866 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzf9\" (UniqueName: \"kubernetes.io/projected/5c438b1f-2443-4f29-9b8c-dab16153b12f-kube-api-access-2nzf9\") pod \"payload-processing-e2e-ipp-a-954773-5974987546-jmjmm\" (UID: \"5c438b1f-2443-4f29-9b8c-dab16153b12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:58:56.278268 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.278245 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c438b1f-2443-4f29-9b8c-dab16153b12f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-954773-5974987546-jmjmm\" (UID: \"5c438b1f-2443-4f29-9b8c-dab16153b12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:58:56.278474 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.278454 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8wt\" (UniqueName: \"kubernetes.io/projected/c1b0120d-39b7-49e7-afe8-658da51a9a0c-kube-api-access-7c8wt\") pod \"payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx\" (UID: \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:58:56.282314 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.282287 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0120d-39b7-49e7-afe8-658da51a9a0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx\" (UID: \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:58:56.332487 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.332378 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8wt\" (UniqueName: \"kubernetes.io/projected/c1b0120d-39b7-49e7-afe8-658da51a9a0c-kube-api-access-7c8wt\") pod \"payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx\" (UID: \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:58:56.382591 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.382545 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzf9\" (UniqueName: \"kubernetes.io/projected/5c438b1f-2443-4f29-9b8c-dab16153b12f-kube-api-access-2nzf9\") pod \"payload-processing-e2e-ipp-a-954773-5974987546-jmjmm\" (UID: \"5c438b1f-2443-4f29-9b8c-dab16153b12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:58:56.383247 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.383221 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c438b1f-2443-4f29-9b8c-dab16153b12f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-954773-5974987546-jmjmm\" (UID: \"5c438b1f-2443-4f29-9b8c-dab16153b12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:58:56.384501 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.384437 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c438b1f-2443-4f29-9b8c-dab16153b12f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-954773-5974987546-jmjmm\" (UID: \"5c438b1f-2443-4f29-9b8c-dab16153b12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:58:56.399610 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.399571 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:58:56.407479 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.407447 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzf9\" (UniqueName: \"kubernetes.io/projected/5c438b1f-2443-4f29-9b8c-dab16153b12f-kube-api-access-2nzf9\") pod \"payload-processing-e2e-ipp-a-954773-5974987546-jmjmm\" (UID: \"5c438b1f-2443-4f29-9b8c-dab16153b12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:58:56.445855 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.445815 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:58:56.716301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.716272 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx"] Oct 30 01:58:56.717736 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:58:56.717706 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b0120d_39b7_49e7_afe8_658da51a9a0c.slice/crio-9349474077cca9482c565b3815765b705b5312157f8f8c6d15755a917f09642b WatchSource:0}: Error finding container 9349474077cca9482c565b3815765b705b5312157f8f8c6d15755a917f09642b: Status 404 returned error can't find the container with id 9349474077cca9482c565b3815765b705b5312157f8f8c6d15755a917f09642b Oct 30 01:58:56.780817 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.780757 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" event={"ID":"c1b0120d-39b7-49e7-afe8-658da51a9a0c","Type":"ContainerStarted","Data":"9349474077cca9482c565b3815765b705b5312157f8f8c6d15755a917f09642b"} Oct 30 01:58:56.816886 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:58:56.816849 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c438b1f_2443_4f29_9b8c_dab16153b12f.slice/crio-16ac5055e8ccf6284071a217401e90beb5fc1cb4a40d5187009cefa930fbf267 WatchSource:0}: Error finding container 16ac5055e8ccf6284071a217401e90beb5fc1cb4a40d5187009cefa930fbf267: Status 404 returned error can't find the container with id 16ac5055e8ccf6284071a217401e90beb5fc1cb4a40d5187009cefa930fbf267 Oct 30 01:58:56.817386 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:56.817358 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm"] Oct 30 01:58:57.791321 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:57.791280 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" event={"ID":"c1b0120d-39b7-49e7-afe8-658da51a9a0c","Type":"ContainerStarted","Data":"e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598"} Oct 30 01:58:57.791831 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:57.791346 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:58:57.793116 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:57.793092 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" event={"ID":"5c438b1f-2443-4f29-9b8c-dab16153b12f","Type":"ContainerStarted","Data":"c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79"} Oct 30 01:58:57.793223 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:57.793133 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" event={"ID":"5c438b1f-2443-4f29-9b8c-dab16153b12f","Type":"ContainerStarted","Data":"16ac5055e8ccf6284071a217401e90beb5fc1cb4a40d5187009cefa930fbf267"} Oct 30 01:58:57.793223 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:57.793155 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:58:57.837728 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:57.837665 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" podStartSLOduration=1.837647958 podStartE2EDuration="1.837647958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:57.834644901 +0000 UTC m=+1816.337577123" watchObservedRunningTime="2026-10-30 01:58:57.837647958 +0000 UTC m=+1816.340580179" Oct 30 01:58:57.889209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:58:57.889151 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" podStartSLOduration=1.889134521 podStartE2EDuration="1.889134521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:57.888496409 +0000 UTC m=+1816.391428631" watchObservedRunningTime="2026-10-30 01:58:57.889134521 +0000 UTC m=+1816.392066741" Oct 30 01:59:03.153712 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:03.153668 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5"] Oct 30 01:59:03.154212 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:03.153957 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" containerID="cri-o://678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a" gracePeriod=30 Oct 30 01:59:06.002622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:06.002544 5035 patch_prober.go:29] interesting pod/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:59:06.003029 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:06.002626 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:59:08.477896 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.477860 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:59:08.667054 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667015 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-token\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-token\") pod \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " Oct 30 01:59:08.667215 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667109 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-certs\") pod \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " Oct 30 01:59:08.667215 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667138 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-envoy\") pod \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " Oct 30 01:59:08.667339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667321 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-data\") pod \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " Oct 30 01:59:08.667405 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667373 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istiod-ca-cert\") pod \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " Oct 30 01:59:08.667448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667435 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-podinfo\") pod \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " Oct 30 01:59:08.667506 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667454 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-certs" pod "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" (UID: "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.667506 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667468 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-kube-api-access-l68bs\") pod \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " Oct 30 01:59:08.667642 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667511 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-credential-socket\") pod \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " Oct 30 01:59:08.667642 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667567 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-data" pod "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" (UID: "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.667740 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667710 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-socket\") pod \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\" (UID: \"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f\") " Oct 30 01:59:08.668149 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667866 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istiod-ca-cert" pod "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" (UID: "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:08.668149 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.667899 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-credential-socket" pod "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" (UID: "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.668149 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.668111 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-socket" pod "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" (UID: "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.668505 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.668480 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.668618 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.668513 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.668618 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.668582 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.668618 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.668600 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.668618 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.668614 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.669629 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.669597 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-envoy" pod "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" (UID: "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:08.669743 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.669731 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-token" pod "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" (UID: "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:08.669797 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.669728 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-podinfo" pod "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" (UID: "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:08.669841 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.669825 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-kube-api-access-l68bs" pod "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" (UID: "289bd9e9-b9a1-46e4-a0ca-2be7fceb543f"). InnerVolumeSpecName "kube-api-access-l68bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:08.770260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.770221 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.770260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.770251 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.770260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.770261 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-kube-api-access-l68bs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.770260 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.770272 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:08.804518 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.804480 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:59:08.804740 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.804712 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:59:08.905859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.899158 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" Oct 30 01:59:08.905859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.898818 5035 generic.go:356] "Generic (PLEG): container finished" podID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerID="678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a" exitCode=0 Oct 30 01:59:08.905859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.899296 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" event={"ID":"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f","Type":"ContainerDied","Data":"678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a"} Oct 30 01:59:08.905859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.899373 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5" event={"ID":"289bd9e9-b9a1-46e4-a0ca-2be7fceb543f","Type":"ContainerDied","Data":"6ba5d3e4ff48517d94d7820a0ce69d6ec6c6e5f55228188ced780e499b8e399d"} Oct 30 01:59:08.905859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.899408 5035 scope.go:122] "RemoveContainer" containerID="678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a" Oct 30 01:59:08.917192 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.917169 5035 scope.go:122] "RemoveContainer" containerID="678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a" Oct 30 01:59:08.917605 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:08.917574 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a\": container with ID starting with 678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a not found: ID does not exist" containerID="678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a" Oct 30 01:59:08.917680 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.917618 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a"} err="failed to get container status \"678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a\": rpc error: code = NotFound desc = could not find container \"678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a\": container with ID starting with 678bcf1302541d3b953e9dd2471900ab43386c52a582811760738c838741209a not found: ID does not exist" Oct 30 01:59:08.998962 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:08.998927 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5"] Oct 30 01:59:09.005885 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:09.005852 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2933ff00-openshift-default-5544ff85d5-hxtd5"] Oct 30 01:59:09.645058 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:09.645021 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" path="/var/lib/kubelet/pods/289bd9e9-b9a1-46e4-a0ca-2be7fceb543f/volumes" Oct 30 01:59:10.673489 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.673451 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w"] Oct 30 01:59:10.674201 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.674180 5035 cpu_manager.go:414] "containerMap: removing container" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" Oct 30 01:59:10.674264 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.674203 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" Oct 30 01:59:10.674381 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.674372 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="289bd9e9-b9a1-46e4-a0ca-2be7fceb543f" containerName="istio-proxy" Oct 30 01:59:10.679363 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.679341 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:10.699060 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.699025 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a06db84-72df-4a11-8164-0353070babca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w\" (UID: \"1a06db84-72df-4a11-8164-0353070babca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:10.699215 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.699090 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnnkw\" (UniqueName: \"kubernetes.io/projected/1a06db84-72df-4a11-8164-0353070babca-kube-api-access-pnnkw\") pod \"payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w\" (UID: \"1a06db84-72df-4a11-8164-0353070babca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:10.717903 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.717869 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w"] Oct 30 01:59:10.741881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.741815 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb"] Oct 30 01:59:10.745581 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.745500 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b273b6\"" Oct 30 01:59:10.748630 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.748598 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:10.773049 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.773006 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b273b6-dockercfg-429fl\"" Oct 30 01:59:10.774644 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.774608 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb"] Oct 30 01:59:10.801606 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.801554 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrmf2\" (UniqueName: \"kubernetes.io/projected/c0082d45-3b3e-446f-829e-1dcb1eb71676-kube-api-access-nrmf2\") pod \"payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb\" (UID: \"c0082d45-3b3e-446f-829e-1dcb1eb71676\") " pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:10.801881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.801751 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a06db84-72df-4a11-8164-0353070babca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w\" (UID: \"1a06db84-72df-4a11-8164-0353070babca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:10.801881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.801816 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnnkw\" (UniqueName: \"kubernetes.io/projected/1a06db84-72df-4a11-8164-0353070babca-kube-api-access-pnnkw\") pod \"payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w\" (UID: \"1a06db84-72df-4a11-8164-0353070babca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:10.801881 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.801855 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0082d45-3b3e-446f-829e-1dcb1eb71676-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb\" (UID: \"c0082d45-3b3e-446f-829e-1dcb1eb71676\") " pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:10.803482 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.803456 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a06db84-72df-4a11-8164-0353070babca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w\" (UID: \"1a06db84-72df-4a11-8164-0353070babca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:10.869102 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.869061 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnnkw\" (UniqueName: \"kubernetes.io/projected/1a06db84-72df-4a11-8164-0353070babca-kube-api-access-pnnkw\") pod \"payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w\" (UID: \"1a06db84-72df-4a11-8164-0353070babca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:10.919660 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.906329 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmf2\" (UniqueName: \"kubernetes.io/projected/c0082d45-3b3e-446f-829e-1dcb1eb71676-kube-api-access-nrmf2\") pod \"payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb\" (UID: \"c0082d45-3b3e-446f-829e-1dcb1eb71676\") " pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:10.919660 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.906909 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0082d45-3b3e-446f-829e-1dcb1eb71676-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb\" (UID: \"c0082d45-3b3e-446f-829e-1dcb1eb71676\") " pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:10.987817 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.969688 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmf2\" (UniqueName: \"kubernetes.io/projected/c0082d45-3b3e-446f-829e-1dcb1eb71676-kube-api-access-nrmf2\") pod \"payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb\" (UID: \"c0082d45-3b3e-446f-829e-1dcb1eb71676\") " pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:10.996941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.996901 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0082d45-3b3e-446f-829e-1dcb1eb71676-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb\" (UID: \"c0082d45-3b3e-446f-829e-1dcb1eb71676\") " pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:10.997606 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:10.997402 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:11.066427 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:11.066132 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:11.349080 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:11.349048 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w"] Oct 30 01:59:11.349761 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:11.349734 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a06db84_72df_4a11_8164_0353070babca.slice/crio-6480dc2316714c652dec4e52cef2daeecec49599ee6bf649161711e0cafac732 WatchSource:0}: Error finding container 6480dc2316714c652dec4e52cef2daeecec49599ee6bf649161711e0cafac732: Status 404 returned error can't find the container with id 6480dc2316714c652dec4e52cef2daeecec49599ee6bf649161711e0cafac732 Oct 30 01:59:11.413241 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:11.413208 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb"] Oct 30 01:59:11.414012 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:11.413976 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0082d45_3b3e_446f_829e_1dcb1eb71676.slice/crio-3dd42969c9727b29710a059f78b6afb56fd67531764bbfd29621e50dafef13ad WatchSource:0}: Error finding container 3dd42969c9727b29710a059f78b6afb56fd67531764bbfd29621e50dafef13ad: Status 404 returned error can't find the container with id 3dd42969c9727b29710a059f78b6afb56fd67531764bbfd29621e50dafef13ad Oct 30 01:59:11.978573 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:11.978518 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" event={"ID":"1a06db84-72df-4a11-8164-0353070babca","Type":"ContainerStarted","Data":"728dda8fbe7b8cdfa56c233379582ce533db2aa9cf57b01f29936782f3e5b1e6"} Oct 30 01:59:11.979097 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:11.978601 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" event={"ID":"1a06db84-72df-4a11-8164-0353070babca","Type":"ContainerStarted","Data":"6480dc2316714c652dec4e52cef2daeecec49599ee6bf649161711e0cafac732"} Oct 30 01:59:11.979097 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:11.978628 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:11.980606 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:11.980577 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" event={"ID":"c0082d45-3b3e-446f-829e-1dcb1eb71676","Type":"ContainerStarted","Data":"15fafa42409166c76b7e9e6b91383f846c5e4e1d5015e4ba239c3fca8c6cc51a"} Oct 30 01:59:11.980730 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:11.980630 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:11.980730 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:11.980647 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" event={"ID":"c0082d45-3b3e-446f-829e-1dcb1eb71676","Type":"ContainerStarted","Data":"3dd42969c9727b29710a059f78b6afb56fd67531764bbfd29621e50dafef13ad"} Oct 30 01:59:12.025079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.025021 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" podStartSLOduration=2.025003765 podStartE2EDuration="2.025003765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:12.021187416 +0000 UTC m=+1830.524119646" watchObservedRunningTime="2026-10-30 01:59:12.025003765 +0000 UTC m=+1830.527936046" Oct 30 01:59:12.075705 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.075644 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" podStartSLOduration=2.075628187 podStartE2EDuration="2.075628187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:12.072646105 +0000 UTC m=+1830.575578338" watchObservedRunningTime="2026-10-30 01:59:12.075628187 +0000 UTC m=+1830.578560406" Oct 30 01:59:12.462294 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.462252 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn"] Oct 30 01:59:12.466978 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.466951 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.486024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.485454 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-954773-serving-cert\"" Oct 30 01:59:12.491759 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.491722 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn"] Oct 30 01:59:12.539414 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.539369 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-metrics-certs\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.539698 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.539647 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cdrz\" (UniqueName: \"kubernetes.io/projected/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-kube-api-access-5cdrz\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.539782 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.539702 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.540014 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.539979 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-maas-api-tls\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.641452 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.641402 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-metrics-certs\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.641687 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.641635 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cdrz\" (UniqueName: \"kubernetes.io/projected/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-kube-api-access-5cdrz\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.641761 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.641684 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.642288 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.642015 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-maas-api-tls\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.643291 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.643259 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.645608 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.645542 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-metrics-certs\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.645993 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.645974 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-maas-api-tls\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.667233 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.667200 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cdrz\" (UniqueName: \"kubernetes.io/projected/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-kube-api-access-5cdrz\") pod \"maas-api-e2e-ipp-a-954773-7d44499fd-s25fn\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:12.784598 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:12.784443 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:13.109979 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:13.109940 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn"] Oct 30 01:59:13.110780 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:13.110756 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2771cfbd_4b0e_4db1_9ea7_c11fc78b4c47.slice/crio-c020fc616b89fa02325ad509a4d285c2d182a813bd33235618ec428438f936c0 WatchSource:0}: Error finding container c020fc616b89fa02325ad509a4d285c2d182a813bd33235618ec428438f936c0: Status 404 returned error can't find the container with id c020fc616b89fa02325ad509a4d285c2d182a813bd33235618ec428438f936c0 Oct 30 01:59:14.007139 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:14.007095 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" event={"ID":"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47","Type":"ContainerStarted","Data":"a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76"} Oct 30 01:59:14.007139 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:14.007168 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" event={"ID":"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47","Type":"ContainerStarted","Data":"c020fc616b89fa02325ad509a4d285c2d182a813bd33235618ec428438f936c0"} Oct 30 01:59:14.007513 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:14.007217 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:14.059612 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:14.059542 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" podStartSLOduration=1.656152195 podStartE2EDuration="2.059511179s" totalImagesPullingTime="403.358984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:14.056146619 +0000 UTC m=+1832.559078839" watchObservedRunningTime="2026-10-30 01:59:14.059511179 +0000 UTC m=+1832.562443401" Oct 30 01:59:16.014671 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.014624 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv"] Oct 30 01:59:16.020412 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.020382 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.074024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.072307 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv"] Oct 30 01:59:16.123772 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.123723 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.123984 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.123786 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8hs9\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-kube-api-access-d8hs9\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.123984 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.123880 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.123984 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.123943 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.124093 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.124051 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.124246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.124229 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.124361 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.124346 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.124400 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.124376 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.124462 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.124450 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.125492 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.125474 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-86dd79-openshift-default-dockercfg-4zq9k\"" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.226189 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8hs9\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-kube-api-access-d8hs9\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.226261 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.226326 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.226393 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.226570 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.226681 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.226714 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.226774 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.226826 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.227262 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.227589 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.227752 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.228031 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.234548 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.228146 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.236055 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.236022 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.242419 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.242229 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.266995 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.258334 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8hs9\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-kube-api-access-d8hs9\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.266995 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.266315 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.392645 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.386841 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:16.739247 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.739208 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv"] Oct 30 01:59:16.741547 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:16.741496 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c118c25_2881_4ec9_a079_fdae4c67f8fd.slice/crio-8bcf54bc3849a6a337eb61e75f00ef2ba27d5322dd0fb42a8589b34c403254bd WatchSource:0}: Error finding container 8bcf54bc3849a6a337eb61e75f00ef2ba27d5322dd0fb42a8589b34c403254bd: Status 404 returned error can't find the container with id 8bcf54bc3849a6a337eb61e75f00ef2ba27d5322dd0fb42a8589b34c403254bd Oct 30 01:59:16.744521 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.744483 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:16.744629 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.744573 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:16.744629 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:16.744608 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:17.034126 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:17.034022 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" event={"ID":"2c118c25-2881-4ec9-a079-fdae4c67f8fd","Type":"ContainerStarted","Data":"b42c92fc1b1d26f61912b82e30612d97e6b7f9d235d489df9b43deb1cd5d7ec0"} Oct 30 01:59:17.034126 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:17.034091 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" event={"ID":"2c118c25-2881-4ec9-a079-fdae4c67f8fd","Type":"ContainerStarted","Data":"8bcf54bc3849a6a337eb61e75f00ef2ba27d5322dd0fb42a8589b34c403254bd"} Oct 30 01:59:17.093171 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:17.093083 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" podStartSLOduration=2.09306577 podStartE2EDuration="2.09306577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:17.087696938 +0000 UTC m=+1835.590629185" watchObservedRunningTime="2026-10-30 01:59:17.09306577 +0000 UTC m=+1835.595997990" Oct 30 01:59:17.388611 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:17.387919 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:17.392552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:17.391445 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 01:59:17.392552 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:17.391600 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 01:59:17.630846 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:17.630806 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:59:18.391079 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:18.391017 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 01:59:18.391689 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:18.391135 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 01:59:19.391555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:19.387990 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 01:59:19.391555 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:19.388056 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 01:59:20.020350 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:20.020318 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 01:59:20.389627 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:20.389027 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 01:59:20.389627 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:20.389102 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 01:59:20.703076 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:20.703037 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw"] Oct 30 01:59:21.392747 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:21.392396 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 01:59:21.392747 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:21.392494 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 01:59:22.388146 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:22.388096 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 01:59:22.388863 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:22.388821 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 01:59:22.993448 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:22.993358 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 01:59:22.993650 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:22.993457 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 01:59:23.005511 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.005473 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l"] Oct 30 01:59:23.010394 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.010373 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:23.037811 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.037713 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l"] Oct 30 01:59:23.037993 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.037918 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6ba1d9\"" Oct 30 01:59:23.064618 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.063266 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6ba1d9-dockercfg-knmhl\"" Oct 30 01:59:23.154554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.153632 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l\" (UID: \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:23.154554 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.153690 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvcv\" (UniqueName: \"kubernetes.io/projected/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-kube-api-access-hcvcv\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l\" (UID: \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:23.167115 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.165014 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss"] Oct 30 01:59:23.234985 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.234880 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:23.240022 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.239867 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss"] Oct 30 01:59:23.262003 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.261967 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l\" (UID: \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:23.262301 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.262277 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvcv\" (UniqueName: \"kubernetes.io/projected/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-kube-api-access-hcvcv\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l\" (UID: \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:23.264033 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.263937 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l\" (UID: \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:23.349812 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.347343 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvcv\" (UniqueName: \"kubernetes.io/projected/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-kube-api-access-hcvcv\") pod \"payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l\" (UID: \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:23.367836 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.366094 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5br7\" (UniqueName: \"kubernetes.io/projected/6b9160fd-4164-484f-b5bb-88f8846934d4-kube-api-access-z5br7\") pod \"payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss\" (UID: \"6b9160fd-4164-484f-b5bb-88f8846934d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:23.367836 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.366413 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9160fd-4164-484f-b5bb-88f8846934d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss\" (UID: \"6b9160fd-4164-484f-b5bb-88f8846934d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:23.444696 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.444660 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:23.445594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.445564 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:23.447316 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.447288 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 01:59:23.467690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.467654 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9160fd-4164-484f-b5bb-88f8846934d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss\" (UID: \"6b9160fd-4164-484f-b5bb-88f8846934d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:23.467916 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.467773 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5br7\" (UniqueName: \"kubernetes.io/projected/6b9160fd-4164-484f-b5bb-88f8846934d4-kube-api-access-z5br7\") pod \"payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss\" (UID: \"6b9160fd-4164-484f-b5bb-88f8846934d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:23.468451 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.468421 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9160fd-4164-484f-b5bb-88f8846934d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss\" (UID: \"6b9160fd-4164-484f-b5bb-88f8846934d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:23.492549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.492489 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5br7\" (UniqueName: \"kubernetes.io/projected/6b9160fd-4164-484f-b5bb-88f8846934d4-kube-api-access-z5br7\") pod \"payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss\" (UID: \"6b9160fd-4164-484f-b5bb-88f8846934d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:23.565483 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.565388 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:23.628687 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.628653 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:23.921935 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.921900 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss"] Oct 30 01:59:23.922325 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:23.922304 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9160fd_4164_484f_b5bb_88f8846934d4.slice/crio-e80c3d67b39d21051eefda1d5f23b0f60ddc2a3aa683afb6bbcb5c8d9118b842 WatchSource:0}: Error finding container e80c3d67b39d21051eefda1d5f23b0f60ddc2a3aa683afb6bbcb5c8d9118b842: Status 404 returned error can't find the container with id e80c3d67b39d21051eefda1d5f23b0f60ddc2a3aa683afb6bbcb5c8d9118b842 Oct 30 01:59:23.981172 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:23.981141 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l"] Oct 30 01:59:23.982924 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:23.982896 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b2d88d_c92d_49e6_84d6_e7267ae2b7fd.slice/crio-44f2a1ba34ef27c0e5d44035759452555161dd903f3be03f71cbe9da23687eff WatchSource:0}: Error finding container 44f2a1ba34ef27c0e5d44035759452555161dd903f3be03f71cbe9da23687eff: Status 404 returned error can't find the container with id 44f2a1ba34ef27c0e5d44035759452555161dd903f3be03f71cbe9da23687eff Oct 30 01:59:24.242878 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:24.242835 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" event={"ID":"6b9160fd-4164-484f-b5bb-88f8846934d4","Type":"ContainerStarted","Data":"ec3bce30a5a5cfa4f2bd7d20ff4b9a45daab62817deddeee648928d9d1c74dae"} Oct 30 01:59:24.243110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:24.242904 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" event={"ID":"6b9160fd-4164-484f-b5bb-88f8846934d4","Type":"ContainerStarted","Data":"e80c3d67b39d21051eefda1d5f23b0f60ddc2a3aa683afb6bbcb5c8d9118b842"} Oct 30 01:59:24.243110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:24.242933 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:24.245110 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:24.245074 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" event={"ID":"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd","Type":"ContainerStarted","Data":"d4f8efd9374f4043a3ba87713d580f6acd7ee4f4c7e619cb4be07268894cf755"} Oct 30 01:59:24.245262 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:24.245136 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" event={"ID":"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd","Type":"ContainerStarted","Data":"44f2a1ba34ef27c0e5d44035759452555161dd903f3be03f71cbe9da23687eff"} Oct 30 01:59:24.245345 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:24.245332 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:24.290294 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:24.290238 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" podStartSLOduration=1.290222563 podStartE2EDuration="1.290222563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:24.286030431 +0000 UTC m=+1842.788962652" watchObservedRunningTime="2026-10-30 01:59:24.290222563 +0000 UTC m=+1842.793154782" Oct 30 01:59:24.331490 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:24.331433 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" podStartSLOduration=2.331415161 podStartE2EDuration="2.331415161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:24.330294984 +0000 UTC m=+1842.833227215" watchObservedRunningTime="2026-10-30 01:59:24.331415161 +0000 UTC m=+1842.834347425" Oct 30 01:59:25.420114 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.420070 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k"] Oct 30 01:59:25.425933 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.425901 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:25.451599 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.451552 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k"] Oct 30 01:59:25.521610 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.513934 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5c90f5\"" Oct 30 01:59:25.607159 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.594212 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqw7z\" (UniqueName: \"kubernetes.io/projected/24af78d5-514c-41c6-b972-ab3b298c69d1-kube-api-access-gqw7z\") pod \"payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k\" (UID: \"24af78d5-514c-41c6-b972-ab3b298c69d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:25.607690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.607656 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24af78d5-514c-41c6-b972-ab3b298c69d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k\" (UID: \"24af78d5-514c-41c6-b972-ab3b298c69d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:25.608190 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.594475 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5c90f5-dockercfg-ss67g\"" Oct 30 01:59:25.704196 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.704151 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" podUID="d469ef37-96c3-434b-9d00-7a8771832df7" containerName="payload-processing" containerID="cri-o://8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d" gracePeriod=25 Oct 30 01:59:25.709706 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.709675 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqw7z\" (UniqueName: \"kubernetes.io/projected/24af78d5-514c-41c6-b972-ab3b298c69d1-kube-api-access-gqw7z\") pod \"payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k\" (UID: \"24af78d5-514c-41c6-b972-ab3b298c69d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:25.709859 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.709844 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24af78d5-514c-41c6-b972-ab3b298c69d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k\" (UID: \"24af78d5-514c-41c6-b972-ab3b298c69d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:25.710658 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.710636 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24af78d5-514c-41c6-b972-ab3b298c69d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k\" (UID: \"24af78d5-514c-41c6-b972-ab3b298c69d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:25.735181 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.735151 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqw7z\" (UniqueName: \"kubernetes.io/projected/24af78d5-514c-41c6-b972-ab3b298c69d1-kube-api-access-gqw7z\") pod \"payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k\" (UID: \"24af78d5-514c-41c6-b972-ab3b298c69d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:25.801252 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:25.801214 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:26.053626 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.053598 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:59:26.116971 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.116927 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d469ef37-96c3-434b-9d00-7a8771832df7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d469ef37-96c3-434b-9d00-7a8771832df7-plugins-config-volume\") pod \"d469ef37-96c3-434b-9d00-7a8771832df7\" (UID: \"d469ef37-96c3-434b-9d00-7a8771832df7\") " Oct 30 01:59:26.117383 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.117351 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d469ef37-96c3-434b-9d00-7a8771832df7-plugins-config-volume" pod "d469ef37-96c3-434b-9d00-7a8771832df7" (UID: "d469ef37-96c3-434b-9d00-7a8771832df7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:26.117662 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.117397 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d469ef37-96c3-434b-9d00-7a8771832df7-kube-api-access-hktmn\" (UniqueName: \"kubernetes.io/projected/d469ef37-96c3-434b-9d00-7a8771832df7-kube-api-access-hktmn\") pod \"d469ef37-96c3-434b-9d00-7a8771832df7\" (UID: \"d469ef37-96c3-434b-9d00-7a8771832df7\") " Oct 30 01:59:26.118197 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.118160 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d469ef37-96c3-434b-9d00-7a8771832df7-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:26.120403 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.120366 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d469ef37-96c3-434b-9d00-7a8771832df7-kube-api-access-hktmn" pod "d469ef37-96c3-434b-9d00-7a8771832df7" (UID: "d469ef37-96c3-434b-9d00-7a8771832df7"). InnerVolumeSpecName "kube-api-access-hktmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:26.143942 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:26.143891 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24af78d5_514c_41c6_b972_ab3b298c69d1.slice/crio-4d4ae4612e4e93cbb5970486e4cf5e104e1ebf3a04bb648251956540883485f6 WatchSource:0}: Error finding container 4d4ae4612e4e93cbb5970486e4cf5e104e1ebf3a04bb648251956540883485f6: Status 404 returned error can't find the container with id 4d4ae4612e4e93cbb5970486e4cf5e104e1ebf3a04bb648251956540883485f6 Oct 30 01:59:26.144658 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.144565 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k"] Oct 30 01:59:26.225325 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.225272 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktmn\" (UniqueName: \"kubernetes.io/projected/d469ef37-96c3-434b-9d00-7a8771832df7-kube-api-access-hktmn\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:26.287748 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.287630 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" event={"ID":"24af78d5-514c-41c6-b972-ab3b298c69d1","Type":"ContainerStarted","Data":"fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d"} Oct 30 01:59:26.287748 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.287704 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" event={"ID":"24af78d5-514c-41c6-b972-ab3b298c69d1","Type":"ContainerStarted","Data":"4d4ae4612e4e93cbb5970486e4cf5e104e1ebf3a04bb648251956540883485f6"} Oct 30 01:59:26.288722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.288678 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:26.290613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.290587 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" Oct 30 01:59:26.290613 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.290606 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" event={"ID":"d469ef37-96c3-434b-9d00-7a8771832df7","Type":"ContainerDied","Data":"8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d"} Oct 30 01:59:26.290808 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.290653 5035 scope.go:122] "RemoveContainer" containerID="8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d" Oct 30 01:59:26.290808 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.290583 5035 generic.go:356] "Generic (PLEG): container finished" podID="d469ef37-96c3-434b-9d00-7a8771832df7" containerID="8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d" exitCode=0 Oct 30 01:59:26.290808 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.290776 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw" event={"ID":"d469ef37-96c3-434b-9d00-7a8771832df7","Type":"ContainerDied","Data":"26fcf9e2c68366fd77b7acab7e7dd0161e59bd95240b48017e9ce9918c9db65c"} Oct 30 01:59:26.306138 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.306107 5035 scope.go:122] "RemoveContainer" containerID="8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d" Oct 30 01:59:26.306840 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:26.306815 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d\": container with ID starting with 8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d not found: ID does not exist" containerID="8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d" Oct 30 01:59:26.306946 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.306853 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d"} err="failed to get container status \"8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d\": rpc error: code = NotFound desc = could not find container \"8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d\": container with ID starting with 8ef31ee4a58b32d6ee0de446089775f216c670f706e9e26521fd91a05d10ed6d not found: ID does not exist" Oct 30 01:59:26.334296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.334229 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" podStartSLOduration=1.334200078 podStartE2EDuration="1.334200078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:26.330908307 +0000 UTC m=+1844.833840526" watchObservedRunningTime="2026-10-30 01:59:26.334200078 +0000 UTC m=+1844.837132300" Oct 30 01:59:26.371893 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.371850 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw"] Oct 30 01:59:26.376068 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:26.376035 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86dd79-57bdfbd4c4-rgsbw"] Oct 30 01:59:27.654702 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:27.654663 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d469ef37-96c3-434b-9d00-7a8771832df7" path="/var/lib/kubelet/pods/d469ef37-96c3-434b-9d00-7a8771832df7/volumes" Oct 30 01:59:35.255477 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:35.255443 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 01:59:35.256144 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:35.255671 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 01:59:37.330105 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:37.330056 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5"] Oct 30 01:59:37.330630 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:37.330410 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" containerID="cri-o://43a868e538322d0c1483f42e9e9567a32b97439a56a28a6d311781bf9bc86309" gracePeriod=30 Oct 30 01:59:37.342040 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:37.341996 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km"] Oct 30 01:59:37.342337 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:37.342273 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" containerID="cri-o://873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5" gracePeriod=30 Oct 30 01:59:37.348220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:37.347552 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh"] Oct 30 01:59:37.348387 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:37.348287 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" containerID="cri-o://9f85789798b64e51aaaf2f4a3016ba88662c41d70cd9c555bdcd08fd1d177aba" gracePeriod=30 Oct 30 01:59:37.358568 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:37.358510 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg"] Oct 30 01:59:37.358858 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:37.358809 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" containerID="cri-o://16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037" gracePeriod=30 Oct 30 01:59:38.311785 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:38.311750 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 01:59:39.048254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:39.048209 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm"] Oct 30 01:59:39.089700 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:39.089651 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx"] Oct 30 01:59:40.823461 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:40.823410 5035 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:59:40.823971 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:40.823500 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:59:42.664408 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.664311 5035 scope.go:122] "RemoveContainer" containerID="1fd42eb5dbe625f99e2d2f911ae1014280e9b120fa5f474f7fb98e6e255ad684" Oct 30 01:59:42.700323 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.700289 5035 scope.go:122] "RemoveContainer" containerID="a9d95ad03efafca066a8cd853537a85d8dc103ee53f867503a0c48add892c22a" Oct 30 01:59:42.714617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.712856 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:59:42.714617 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.714471 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:59:42.729977 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.729846 5035 scope.go:122] "RemoveContainer" containerID="9f85789798b64e51aaaf2f4a3016ba88662c41d70cd9c555bdcd08fd1d177aba" Oct 30 01:59:42.744814 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.744716 5035 scope.go:122] "RemoveContainer" containerID="cb56ccc9a615c43183eecd359cc5d62f58caaf0e4cb7e33810cc114ed79fd261" Oct 30 01:59:42.758944 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.758915 5035 scope.go:122] "RemoveContainer" containerID="43a868e538322d0c1483f42e9e9567a32b97439a56a28a6d311781bf9bc86309" Oct 30 01:59:42.772255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.772227 5035 scope.go:122] "RemoveContainer" containerID="7f0fd15f37ce62331c22703670efe4d7342f3db5209ca013c89c05b1102b958c" Oct 30 01:59:42.804132 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.804107 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:59:42.811509 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.811484 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:59:42.868992 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.868955 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-credential-socket\") pod \"e3b88633-6d70-42e1-8731-bf1024b67844\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " Oct 30 01:59:42.869143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869014 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-data\") pod \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " Oct 30 01:59:42.869143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869074 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-kube-api-access-dq5fv\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-kube-api-access-dq5fv\") pod \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " Oct 30 01:59:42.869143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869125 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-envoy\") pod \"e3b88633-6d70-42e1-8731-bf1024b67844\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " Oct 30 01:59:42.869300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869152 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-token\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-token\") pod \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " Oct 30 01:59:42.869300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869182 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-envoy\") pod \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " Oct 30 01:59:42.869300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869227 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-podinfo\") pod \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " Oct 30 01:59:42.869300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869266 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-credential-socket" pod "e3b88633-6d70-42e1-8731-bf1024b67844" (UID: "e3b88633-6d70-42e1-8731-bf1024b67844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.869300 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869282 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-socket\") pod \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " Oct 30 01:59:42.869570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869302 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-data" pod "4727e030-5504-4bc5-85f0-c9d26fc12aa9" (UID: "4727e030-5504-4bc5-85f0-c9d26fc12aa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.869570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869314 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-kube-api-access-n2r22\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-kube-api-access-n2r22\") pod \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " Oct 30 01:59:42.869570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869349 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-podinfo\") pod \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " Oct 30 01:59:42.869570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869410 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-data\") pod \"e3b88633-6d70-42e1-8731-bf1024b67844\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " Oct 30 01:59:42.869570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869440 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-certs\") pod \"e3b88633-6d70-42e1-8731-bf1024b67844\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " Oct 30 01:59:42.869570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869466 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-istio-token\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-istio-token\") pod \"e3b88633-6d70-42e1-8731-bf1024b67844\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " Oct 30 01:59:42.869570 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869513 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-credential-socket\") pod \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " Oct 30 01:59:42.869915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869585 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3b88633-6d70-42e1-8731-bf1024b67844-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b88633-6d70-42e1-8731-bf1024b67844-istio-podinfo\") pod \"e3b88633-6d70-42e1-8731-bf1024b67844\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " Oct 30 01:59:42.869915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869611 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-certs\") pod \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " Oct 30 01:59:42.869915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869638 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-kube-api-access-2gx8g\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-kube-api-access-2gx8g\") pod \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " Oct 30 01:59:42.869915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869678 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-socket\") pod \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " Oct 30 01:59:42.869915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869719 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-podinfo\") pod \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " Oct 30 01:59:42.869915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869769 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-kube-api-access-hmcxj\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-kube-api-access-hmcxj\") pod \"e3b88633-6d70-42e1-8731-bf1024b67844\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " Oct 30 01:59:42.869915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869804 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istiod-ca-cert\") pod \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " Oct 30 01:59:42.869915 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869865 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-certs\") pod \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869929 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-token\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-token\") pod \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.869980 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-envoy\") pod \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870003 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-credential-socket\") pod \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870043 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-credential-socket\") pod \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870063 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-certs" pod "e3b88633-6d70-42e1-8731-bf1024b67844" (UID: "e3b88633-6d70-42e1-8731-bf1024b67844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870091 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-data\") pod \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870124 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-data\") pod \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870182 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-token\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-token\") pod \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870217 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-certs\") pod \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " Oct 30 01:59:42.870290 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870265 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-socket\") pod \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\" (UID: \"36a5121f-31c1-4f5a-a278-9cf5146b78ec\") " Oct 30 01:59:42.870810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870313 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-envoy\") pod \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " Oct 30 01:59:42.870810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870331 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-socket" pod "4727e030-5504-4bc5-85f0-c9d26fc12aa9" (UID: "4727e030-5504-4bc5-85f0-c9d26fc12aa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.870810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870369 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istiod-ca-cert\") pod \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\" (UID: \"b6e47036-a11e-4f4f-90fd-c36e6d7198d2\") " Oct 30 01:59:42.870810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870405 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b88633-6d70-42e1-8731-bf1024b67844-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b88633-6d70-42e1-8731-bf1024b67844-istiod-ca-cert\") pod \"e3b88633-6d70-42e1-8731-bf1024b67844\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " Oct 30 01:59:42.870810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870447 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istiod-ca-cert\") pod \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\" (UID: \"4727e030-5504-4bc5-85f0-c9d26fc12aa9\") " Oct 30 01:59:42.870810 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.870486 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-socket\") pod \"e3b88633-6d70-42e1-8731-bf1024b67844\" (UID: \"e3b88633-6d70-42e1-8731-bf1024b67844\") " Oct 30 01:59:42.871401 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.871376 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.871466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.871409 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.871466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.871423 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.871466 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.871436 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.871694 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.871675 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-socket" pod "e3b88633-6d70-42e1-8731-bf1024b67844" (UID: "e3b88633-6d70-42e1-8731-bf1024b67844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.871939 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.871917 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-credential-socket" pod "4727e030-5504-4bc5-85f0-c9d26fc12aa9" (UID: "4727e030-5504-4bc5-85f0-c9d26fc12aa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.872125 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.872101 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-credential-socket" pod "b6e47036-a11e-4f4f-90fd-c36e6d7198d2" (UID: "b6e47036-a11e-4f4f-90fd-c36e6d7198d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.872298 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.872123 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-envoy" pod "e3b88633-6d70-42e1-8731-bf1024b67844" (UID: "e3b88633-6d70-42e1-8731-bf1024b67844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.872395 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.872157 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-token" pod "b6e47036-a11e-4f4f-90fd-c36e6d7198d2" (UID: "b6e47036-a11e-4f4f-90fd-c36e6d7198d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:42.872482 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.872324 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-socket" pod "b6e47036-a11e-4f4f-90fd-c36e6d7198d2" (UID: "b6e47036-a11e-4f4f-90fd-c36e6d7198d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.872864 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.872827 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istiod-ca-cert" pod "36a5121f-31c1-4f5a-a278-9cf5146b78ec" (UID: "36a5121f-31c1-4f5a-a278-9cf5146b78ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:42.873088 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.873064 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-data" pod "36a5121f-31c1-4f5a-a278-9cf5146b78ec" (UID: "36a5121f-31c1-4f5a-a278-9cf5146b78ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.873255 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.873237 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-data" pod "b6e47036-a11e-4f4f-90fd-c36e6d7198d2" (UID: "b6e47036-a11e-4f4f-90fd-c36e6d7198d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.874572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.874504 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-envoy" pod "b6e47036-a11e-4f4f-90fd-c36e6d7198d2" (UID: "b6e47036-a11e-4f4f-90fd-c36e6d7198d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.874769 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.874726 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-certs" pod "b6e47036-a11e-4f4f-90fd-c36e6d7198d2" (UID: "b6e47036-a11e-4f4f-90fd-c36e6d7198d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.875721 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.875692 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-podinfo" pod "b6e47036-a11e-4f4f-90fd-c36e6d7198d2" (UID: "b6e47036-a11e-4f4f-90fd-c36e6d7198d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:42.875823 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.875700 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-podinfo" pod "4727e030-5504-4bc5-85f0-c9d26fc12aa9" (UID: "4727e030-5504-4bc5-85f0-c9d26fc12aa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:42.875823 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.875725 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-token" pod "36a5121f-31c1-4f5a-a278-9cf5146b78ec" (UID: "36a5121f-31c1-4f5a-a278-9cf5146b78ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:42.876022 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.875968 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-certs" pod "36a5121f-31c1-4f5a-a278-9cf5146b78ec" (UID: "36a5121f-31c1-4f5a-a278-9cf5146b78ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.876168 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.876147 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-socket" pod "36a5121f-31c1-4f5a-a278-9cf5146b78ec" (UID: "36a5121f-31c1-4f5a-a278-9cf5146b78ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.876508 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.876318 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-istio-token" pod "e3b88633-6d70-42e1-8731-bf1024b67844" (UID: "e3b88633-6d70-42e1-8731-bf1024b67844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:42.876656 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.876560 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-certs" pod "4727e030-5504-4bc5-85f0-c9d26fc12aa9" (UID: "4727e030-5504-4bc5-85f0-c9d26fc12aa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.877512 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.877487 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-kube-api-access-dq5fv" pod "4727e030-5504-4bc5-85f0-c9d26fc12aa9" (UID: "4727e030-5504-4bc5-85f0-c9d26fc12aa9"). InnerVolumeSpecName "kube-api-access-dq5fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:42.878067 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.878044 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3b88633-6d70-42e1-8731-bf1024b67844-istio-podinfo" pod "e3b88633-6d70-42e1-8731-bf1024b67844" (UID: "e3b88633-6d70-42e1-8731-bf1024b67844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:42.878286 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.878261 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-credential-socket" pod "36a5121f-31c1-4f5a-a278-9cf5146b78ec" (UID: "36a5121f-31c1-4f5a-a278-9cf5146b78ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.878387 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.878298 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-kube-api-access-hmcxj" pod "e3b88633-6d70-42e1-8731-bf1024b67844" (UID: "e3b88633-6d70-42e1-8731-bf1024b67844"). InnerVolumeSpecName "kube-api-access-hmcxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:42.878744 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.878668 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b88633-6d70-42e1-8731-bf1024b67844-istiod-ca-cert" pod "e3b88633-6d70-42e1-8731-bf1024b67844" (UID: "e3b88633-6d70-42e1-8731-bf1024b67844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:42.878744 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.878696 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istiod-ca-cert" pod "b6e47036-a11e-4f4f-90fd-c36e6d7198d2" (UID: "b6e47036-a11e-4f4f-90fd-c36e6d7198d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:42.878744 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.878704 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istiod-ca-cert" pod "4727e030-5504-4bc5-85f0-c9d26fc12aa9" (UID: "4727e030-5504-4bc5-85f0-c9d26fc12aa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:42.879109 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.878994 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-envoy" pod "4727e030-5504-4bc5-85f0-c9d26fc12aa9" (UID: "4727e030-5504-4bc5-85f0-c9d26fc12aa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.879109 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.879009 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-data" pod "e3b88633-6d70-42e1-8731-bf1024b67844" (UID: "e3b88633-6d70-42e1-8731-bf1024b67844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.879396 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.879371 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-kube-api-access-n2r22" pod "36a5121f-31c1-4f5a-a278-9cf5146b78ec" (UID: "36a5121f-31c1-4f5a-a278-9cf5146b78ec"). InnerVolumeSpecName "kube-api-access-n2r22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:42.880150 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.880126 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-envoy" pod "36a5121f-31c1-4f5a-a278-9cf5146b78ec" (UID: "36a5121f-31c1-4f5a-a278-9cf5146b78ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:42.880889 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.880868 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-token" pod "4727e030-5504-4bc5-85f0-c9d26fc12aa9" (UID: "4727e030-5504-4bc5-85f0-c9d26fc12aa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:42.880967 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.880901 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-kube-api-access-2gx8g" pod "b6e47036-a11e-4f4f-90fd-c36e6d7198d2" (UID: "b6e47036-a11e-4f4f-90fd-c36e6d7198d2"). InnerVolumeSpecName "kube-api-access-2gx8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:42.882437 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.882412 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-podinfo" pod "36a5121f-31c1-4f5a-a278-9cf5146b78ec" (UID: "36a5121f-31c1-4f5a-a278-9cf5146b78ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:42.973637 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.973584 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.973970 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.973948 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.974174 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.974142 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.974305 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.974292 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.974446 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.974433 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.974571 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.974558 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.974678 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.974665 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.974766 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.974755 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.974877 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.974844 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b88633-6d70-42e1-8731-bf1024b67844-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975011 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.974998 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975159 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975143 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975299 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975284 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5fv\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-kube-api-access-dq5fv\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975431 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975404 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975537 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975558 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975572 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975585 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2r22\" (UniqueName: \"kubernetes.io/projected/36a5121f-31c1-4f5a-a278-9cf5146b78ec-kube-api-access-n2r22\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975599 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975613 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b88633-6d70-42e1-8731-bf1024b67844-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975626 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975638 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a5121f-31c1-4f5a-a278-9cf5146b78ec-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975650 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b88633-6d70-42e1-8731-bf1024b67844-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975662 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975676 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gx8g\" (UniqueName: \"kubernetes.io/projected/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-kube-api-access-2gx8g\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.975692 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975691 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.976602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975705 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.976602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975718 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmcxj\" (UniqueName: \"kubernetes.io/projected/e3b88633-6d70-42e1-8731-bf1024b67844-kube-api-access-hmcxj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.976602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975730 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a5121f-31c1-4f5a-a278-9cf5146b78ec-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.976602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975742 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.976602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975754 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4727e030-5504-4bc5-85f0-c9d26fc12aa9-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.976602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975766 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6e47036-a11e-4f4f-90fd-c36e6d7198d2-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:42.976602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:42.975779 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4727e030-5504-4bc5-85f0-c9d26fc12aa9-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:43.450451 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.450410 5035 generic.go:356] "Generic (PLEG): container finished" podID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerID="16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037" exitCode=0 Oct 30 01:59:43.450693 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.450499 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" Oct 30 01:59:43.450693 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.450504 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" event={"ID":"b6e47036-a11e-4f4f-90fd-c36e6d7198d2","Type":"ContainerDied","Data":"16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037"} Oct 30 01:59:43.450693 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.450671 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg" event={"ID":"b6e47036-a11e-4f4f-90fd-c36e6d7198d2","Type":"ContainerDied","Data":"94c00015a045c174ff2da43ba68b579f0543b7016f8768f4bf2ff649188c3396"} Oct 30 01:59:43.450912 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.450706 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" event={"ID":"e3b88633-6d70-42e1-8731-bf1024b67844","Type":"ContainerDied","Data":"43a868e538322d0c1483f42e9e9567a32b97439a56a28a6d311781bf9bc86309"} Oct 30 01:59:43.450912 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.450732 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" event={"ID":"e3b88633-6d70-42e1-8731-bf1024b67844","Type":"ContainerDied","Data":"56ca622c6e40322164ec6386f59829379508fa2967dbe8ac95b6523bac7d1d4b"} Oct 30 01:59:43.450912 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.450759 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" event={"ID":"36a5121f-31c1-4f5a-a278-9cf5146b78ec","Type":"ContainerDied","Data":"9f85789798b64e51aaaf2f4a3016ba88662c41d70cd9c555bdcd08fd1d177aba"} Oct 30 01:59:43.450912 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.450785 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" event={"ID":"36a5121f-31c1-4f5a-a278-9cf5146b78ec","Type":"ContainerDied","Data":"38a6738633d677717850327920e7d82a7230a476c0ef00fd6df204f8789d7cbc"} Oct 30 01:59:43.450912 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.450800 5035 scope.go:122] "RemoveContainer" containerID="16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037" Oct 30 01:59:43.459762 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.459718 5035 generic.go:356] "Generic (PLEG): container finished" podID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerID="873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5" exitCode=0 Oct 30 01:59:43.459941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.459811 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" Oct 30 01:59:43.459941 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.459876 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" event={"ID":"4727e030-5504-4bc5-85f0-c9d26fc12aa9","Type":"ContainerDied","Data":"873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5"} Oct 30 01:59:43.460099 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.459944 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km" event={"ID":"4727e030-5504-4bc5-85f0-c9d26fc12aa9","Type":"ContainerDied","Data":"7db3fbf33e18cd36520e5e8744307384ec738fa299d72dae38d158a67b0af49a"} Oct 30 01:59:43.460296 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.460276 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh" Oct 30 01:59:43.460438 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.460417 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5" Oct 30 01:59:43.467641 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.467624 5035 scope.go:122] "RemoveContainer" containerID="16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037" Oct 30 01:59:43.468162 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:43.468010 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037\": container with ID starting with 16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037 not found: ID does not exist" containerID="16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037" Oct 30 01:59:43.468162 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.468056 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037"} err="failed to get container status \"16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037\": rpc error: code = NotFound desc = could not find container \"16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037\": container with ID starting with 16bee8eb2206df045050ec427a1549f2d80f791b6d9aadd65f1d3b7dfd4ef037 not found: ID does not exist" Oct 30 01:59:43.468162 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.468077 5035 scope.go:122] "RemoveContainer" containerID="873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5" Oct 30 01:59:43.481687 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.481663 5035 scope.go:122] "RemoveContainer" containerID="873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5" Oct 30 01:59:43.481922 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:43.481894 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5\": container with ID starting with 873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5 not found: ID does not exist" containerID="873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5" Oct 30 01:59:43.481991 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.481938 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5"} err="failed to get container status \"873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5\": rpc error: code = NotFound desc = could not find container \"873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5\": container with ID starting with 873cf7d434a12ca3161c3f6b5daeeb3e8e60ed2f441a111c4e92adee42996ad5 not found: ID does not exist" Oct 30 01:59:43.628711 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.628618 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg"] Oct 30 01:59:43.646490 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.646455 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-82nzg"] Oct 30 01:59:43.683323 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.683285 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5"] Oct 30 01:59:43.687227 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.687195 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-n5lt5"] Oct 30 01:59:43.733460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.733425 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh"] Oct 30 01:59:43.737231 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.737205 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-v86jh"] Oct 30 01:59:43.791616 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.791584 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km"] Oct 30 01:59:43.804136 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:43.804087 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-b58km"] Oct 30 01:59:44.048937 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.048887 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" podUID="5c438b1f-2443-4f29-9b8c-dab16153b12f" containerName="payload-processing" containerID="cri-o://c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79" gracePeriod=25 Oct 30 01:59:44.090689 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.090640 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" podUID="c1b0120d-39b7-49e7-afe8-658da51a9a0c" containerName="payload-pre-processing" containerID="cri-o://e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598" gracePeriod=25 Oct 30 01:59:44.404602 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.404512 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:59:44.455476 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.455443 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:59:44.470993 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.470951 5035 generic.go:356] "Generic (PLEG): container finished" podID="5c438b1f-2443-4f29-9b8c-dab16153b12f" containerID="c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79" exitCode=0 Oct 30 01:59:44.470993 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.470996 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" Oct 30 01:59:44.471294 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.471035 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" event={"ID":"5c438b1f-2443-4f29-9b8c-dab16153b12f","Type":"ContainerDied","Data":"c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79"} Oct 30 01:59:44.471294 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.471095 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm" event={"ID":"5c438b1f-2443-4f29-9b8c-dab16153b12f","Type":"ContainerDied","Data":"16ac5055e8ccf6284071a217401e90beb5fc1cb4a40d5187009cefa930fbf267"} Oct 30 01:59:44.471294 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.471130 5035 scope.go:122] "RemoveContainer" containerID="c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79" Oct 30 01:59:44.475209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.475179 5035 generic.go:356] "Generic (PLEG): container finished" podID="c1b0120d-39b7-49e7-afe8-658da51a9a0c" containerID="e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598" exitCode=0 Oct 30 01:59:44.475355 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.475236 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" Oct 30 01:59:44.475355 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.475306 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" event={"ID":"c1b0120d-39b7-49e7-afe8-658da51a9a0c","Type":"ContainerDied","Data":"e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598"} Oct 30 01:59:44.475487 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.475377 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx" event={"ID":"c1b0120d-39b7-49e7-afe8-658da51a9a0c","Type":"ContainerDied","Data":"9349474077cca9482c565b3815765b705b5312157f8f8c6d15755a917f09642b"} Oct 30 01:59:44.491854 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.491825 5035 scope.go:122] "RemoveContainer" containerID="c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79" Oct 30 01:59:44.492272 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:44.492242 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79\": container with ID starting with c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79 not found: ID does not exist" containerID="c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79" Oct 30 01:59:44.492377 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.492280 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79"} err="failed to get container status \"c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79\": rpc error: code = NotFound desc = could not find container \"c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79\": container with ID starting with c90a8f37e7e40ee74e6ab57f8eca120b22e468001df0465db7a0b87263e7ca79 not found: ID does not exist" Oct 30 01:59:44.492377 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.492300 5035 scope.go:122] "RemoveContainer" containerID="e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598" Oct 30 01:59:44.498604 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.498574 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c438b1f-2443-4f29-9b8c-dab16153b12f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c438b1f-2443-4f29-9b8c-dab16153b12f-plugins-config-volume\") pod \"5c438b1f-2443-4f29-9b8c-dab16153b12f\" (UID: \"5c438b1f-2443-4f29-9b8c-dab16153b12f\") " Oct 30 01:59:44.498713 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.498680 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c438b1f-2443-4f29-9b8c-dab16153b12f-kube-api-access-2nzf9\" (UniqueName: \"kubernetes.io/projected/5c438b1f-2443-4f29-9b8c-dab16153b12f-kube-api-access-2nzf9\") pod \"5c438b1f-2443-4f29-9b8c-dab16153b12f\" (UID: \"5c438b1f-2443-4f29-9b8c-dab16153b12f\") " Oct 30 01:59:44.499008 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.498981 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c438b1f-2443-4f29-9b8c-dab16153b12f-plugins-config-volume" pod "5c438b1f-2443-4f29-9b8c-dab16153b12f" (UID: "5c438b1f-2443-4f29-9b8c-dab16153b12f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:44.499329 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.499298 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c438b1f-2443-4f29-9b8c-dab16153b12f-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:44.501142 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.501114 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c438b1f-2443-4f29-9b8c-dab16153b12f-kube-api-access-2nzf9" pod "5c438b1f-2443-4f29-9b8c-dab16153b12f" (UID: "5c438b1f-2443-4f29-9b8c-dab16153b12f"). InnerVolumeSpecName "kube-api-access-2nzf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:44.508085 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.507695 5035 scope.go:122] "RemoveContainer" containerID="e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598" Oct 30 01:59:44.508085 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:44.508072 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598\": container with ID starting with e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598 not found: ID does not exist" containerID="e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598" Oct 30 01:59:44.508243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.508108 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598"} err="failed to get container status \"e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598\": rpc error: code = NotFound desc = could not find container \"e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598\": container with ID starting with e3104bcfda297a03fb8613a80de88c981f8d9c75edb50402911e31c13b281598 not found: ID does not exist" Oct 30 01:59:44.600087 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.599990 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b0120d-39b7-49e7-afe8-658da51a9a0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0120d-39b7-49e7-afe8-658da51a9a0c-plugins-config-volume\") pod \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\" (UID: \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\") " Oct 30 01:59:44.600289 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.600266 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b0120d-39b7-49e7-afe8-658da51a9a0c-kube-api-access-7c8wt\" (UniqueName: \"kubernetes.io/projected/c1b0120d-39b7-49e7-afe8-658da51a9a0c-kube-api-access-7c8wt\") pod \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\" (UID: \"c1b0120d-39b7-49e7-afe8-658da51a9a0c\") " Oct 30 01:59:44.600474 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.600450 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b0120d-39b7-49e7-afe8-658da51a9a0c-plugins-config-volume" pod "c1b0120d-39b7-49e7-afe8-658da51a9a0c" (UID: "c1b0120d-39b7-49e7-afe8-658da51a9a0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:44.600979 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.600960 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nzf9\" (UniqueName: \"kubernetes.io/projected/5c438b1f-2443-4f29-9b8c-dab16153b12f-kube-api-access-2nzf9\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:44.601021 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.600982 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b0120d-39b7-49e7-afe8-658da51a9a0c-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:44.602629 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.602590 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b0120d-39b7-49e7-afe8-658da51a9a0c-kube-api-access-7c8wt" pod "c1b0120d-39b7-49e7-afe8-658da51a9a0c" (UID: "c1b0120d-39b7-49e7-afe8-658da51a9a0c"). InnerVolumeSpecName "kube-api-access-7c8wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:44.702419 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.702375 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8wt\" (UniqueName: \"kubernetes.io/projected/c1b0120d-39b7-49e7-afe8-658da51a9a0c-kube-api-access-7c8wt\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:44.823061 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.823019 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm"] Oct 30 01:59:44.835581 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.835544 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-954773-5974987546-jmjmm"] Oct 30 01:59:44.867006 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.866915 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx"] Oct 30 01:59:44.871969 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:44.871939 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-954773-6b46c87747-zb7xx"] Oct 30 01:59:45.065040 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.064996 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5"] Oct 30 01:59:45.066504 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066470 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d469ef37-96c3-434b-9d00-7a8771832df7" containerName="payload-processing" Oct 30 01:59:45.066504 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066501 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d469ef37-96c3-434b-9d00-7a8771832df7" containerName="payload-processing" Oct 30 01:59:45.066690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066518 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" Oct 30 01:59:45.066690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066549 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" Oct 30 01:59:45.066690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066566 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c1b0120d-39b7-49e7-afe8-658da51a9a0c" containerName="payload-pre-processing" Oct 30 01:59:45.066690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066576 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b0120d-39b7-49e7-afe8-658da51a9a0c" containerName="payload-pre-processing" Oct 30 01:59:45.066690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066599 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" Oct 30 01:59:45.066690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066606 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" Oct 30 01:59:45.066690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066675 5035 cpu_manager.go:414] "containerMap: removing container" podUID="5c438b1f-2443-4f29-9b8c-dab16153b12f" containerName="payload-processing" Oct 30 01:59:45.066690 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066683 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c438b1f-2443-4f29-9b8c-dab16153b12f" containerName="payload-processing" Oct 30 01:59:45.067068 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066743 5035 cpu_manager.go:414] "containerMap: removing container" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" Oct 30 01:59:45.067068 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066750 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" Oct 30 01:59:45.067068 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066760 5035 cpu_manager.go:414] "containerMap: removing container" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" Oct 30 01:59:45.067068 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066767 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" Oct 30 01:59:45.067068 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066812 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" Oct 30 01:59:45.067068 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.066821 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" Oct 30 01:59:45.067320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.067185 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" containerName="istio-proxy" Oct 30 01:59:45.067320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.067211 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" containerName="istio-proxy" Oct 30 01:59:45.067320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.067234 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" containerName="istio-proxy" Oct 30 01:59:45.067320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.067252 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d469ef37-96c3-434b-9d00-7a8771832df7" containerName="payload-processing" Oct 30 01:59:45.067320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.067263 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" Oct 30 01:59:45.067320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.067282 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b0120d-39b7-49e7-afe8-658da51a9a0c" containerName="payload-pre-processing" Oct 30 01:59:45.067320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.067300 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" containerName="istio-proxy" Oct 30 01:59:45.067320 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.067321 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c438b1f-2443-4f29-9b8c-dab16153b12f" containerName="payload-processing" Oct 30 01:59:45.074219 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.074191 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.090161 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.090129 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5"] Oct 30 01:59:45.092889 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.092861 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6ba1d9-serving-cert\"" Oct 30 01:59:45.209622 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.209576 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-metrics-certs\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.209795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.209738 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90193f8e-9356-41c1-b6ba-8e465cdeead7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.209914 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.209894 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.209980 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.209945 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhdsb\" (UniqueName: \"kubernetes.io/projected/90193f8e-9356-41c1-b6ba-8e465cdeead7-kube-api-access-fhdsb\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.311496 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.311442 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90193f8e-9356-41c1-b6ba-8e465cdeead7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.311707 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.311587 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.311707 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.311620 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhdsb\" (UniqueName: \"kubernetes.io/projected/90193f8e-9356-41c1-b6ba-8e465cdeead7-kube-api-access-fhdsb\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.311785 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:45.311735 5035 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-serving-cert: secret "maas-api-e2e-ipp-b-6ba1d9-serving-cert" not found Oct 30 01:59:45.311818 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:45.311810 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls podName:90193f8e-9356-41c1-b6ba-8e465cdeead7 nodeName:}" failed. No retries permitted until 2026-10-30 01:59:45.811787081 +0000 UTC m=+1864.314719296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls") pod "maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" (UID: "90193f8e-9356-41c1-b6ba-8e465cdeead7") : secret "maas-api-e2e-ipp-b-6ba1d9-serving-cert" not found Oct 30 01:59:45.311863 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.311825 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-metrics-certs\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.312262 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.312238 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90193f8e-9356-41c1-b6ba-8e465cdeead7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.315026 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.314996 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-metrics-certs\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.343083 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.343035 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhdsb\" (UniqueName: \"kubernetes.io/projected/90193f8e-9356-41c1-b6ba-8e465cdeead7-kube-api-access-fhdsb\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.645126 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.645037 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a5121f-31c1-4f5a-a278-9cf5146b78ec" path="/var/lib/kubelet/pods/36a5121f-31c1-4f5a-a278-9cf5146b78ec/volumes" Oct 30 01:59:45.645611 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.645596 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4727e030-5504-4bc5-85f0-c9d26fc12aa9" path="/var/lib/kubelet/pods/4727e030-5504-4bc5-85f0-c9d26fc12aa9/volumes" Oct 30 01:59:45.646016 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.646003 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c438b1f-2443-4f29-9b8c-dab16153b12f" path="/var/lib/kubelet/pods/5c438b1f-2443-4f29-9b8c-dab16153b12f/volumes" Oct 30 01:59:45.646839 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.646824 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e47036-a11e-4f4f-90fd-c36e6d7198d2" path="/var/lib/kubelet/pods/b6e47036-a11e-4f4f-90fd-c36e6d7198d2/volumes" Oct 30 01:59:45.647209 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.647197 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b0120d-39b7-49e7-afe8-658da51a9a0c" path="/var/lib/kubelet/pods/c1b0120d-39b7-49e7-afe8-658da51a9a0c/volumes" Oct 30 01:59:45.647670 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.647655 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b88633-6d70-42e1-8731-bf1024b67844" path="/var/lib/kubelet/pods/e3b88633-6d70-42e1-8731-bf1024b67844/volumes" Oct 30 01:59:45.819150 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.819112 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.822146 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.822116 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:45.997760 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:45.997721 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:46.101844 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.101799 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87"] Oct 30 01:59:46.111063 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.111024 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.154965 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.154931 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87"] Oct 30 01:59:46.226718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.226156 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvkfs\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-kube-api-access-vvkfs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.226718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.226210 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.226718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.226237 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.226718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.226431 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.226718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.226474 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.226718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.226523 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.226718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.226631 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.226718 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.226663 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.227521 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.227355 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.323249 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.323214 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5"] Oct 30 01:59:46.325480 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:46.325440 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90193f8e_9356_41c1_b6ba_8e465cdeead7.slice/crio-fd67b3f3ab5729b2e1460781b5f041116c754a1d1aa24d20d5740e7ae1a6adad WatchSource:0}: Error finding container fd67b3f3ab5729b2e1460781b5f041116c754a1d1aa24d20d5740e7ae1a6adad: Status 404 returned error can't find the container with id fd67b3f3ab5729b2e1460781b5f041116c754a1d1aa24d20d5740e7ae1a6adad Oct 30 01:59:46.327460 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.327437 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:59:46.328811 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.328778 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.328966 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.328883 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkfs\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-kube-api-access-vvkfs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.328966 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.328914 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.328966 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.328938 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.329185 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.329095 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.329185 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.329128 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.329185 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.329158 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.329333 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.329193 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.329333 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.329210 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.331408 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.329937 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.331408 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.330610 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.331698 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.331447 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.332584 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.332556 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.334024 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.334000 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.334246 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.334209 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.334708 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.334686 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.362805 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.362772 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.364719 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.364691 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkfs\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-kube-api-access-vvkfs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.428929 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.428894 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:46.502101 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.502065 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" event={"ID":"90193f8e-9356-41c1-b6ba-8e465cdeead7","Type":"ContainerStarted","Data":"fd67b3f3ab5729b2e1460781b5f041116c754a1d1aa24d20d5740e7ae1a6adad"} Oct 30 01:59:46.766099 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.766064 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87"] Oct 30 01:59:46.768574 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:46.768543 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0b842e_01f5_4bb9_9dcb_efc9577a6e45.slice/crio-1a2cf063787f74644e976d23d05e6383e5833206da9d80dddaf9246ef9d937bc WatchSource:0}: Error finding container 1a2cf063787f74644e976d23d05e6383e5833206da9d80dddaf9246ef9d937bc: Status 404 returned error can't find the container with id 1a2cf063787f74644e976d23d05e6383e5833206da9d80dddaf9246ef9d937bc Oct 30 01:59:46.772176 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.772133 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:46.772262 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.772246 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:46.772307 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.772294 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:46.880806 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.880767 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv"] Oct 30 01:59:46.885681 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.885648 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:46.908134 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.908064 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5c90f5-serving-cert\"" Oct 30 01:59:46.913482 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:46.913445 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv"] Oct 30 01:59:47.044063 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.044021 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.044063 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.044068 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-metrics-certs\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.044368 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.044336 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmv9\" (UniqueName: \"kubernetes.io/projected/4eac47eb-462c-49dd-8d15-cd6f90dac721-kube-api-access-kqmv9\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.044573 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.044504 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eac47eb-462c-49dd-8d15-cd6f90dac721-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.146339 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.146293 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.146686 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.146653 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-metrics-certs\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.146914 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.146880 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmv9\" (UniqueName: \"kubernetes.io/projected/4eac47eb-462c-49dd-8d15-cd6f90dac721-kube-api-access-kqmv9\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.147022 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.147002 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eac47eb-462c-49dd-8d15-cd6f90dac721-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.151549 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.148051 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eac47eb-462c-49dd-8d15-cd6f90dac721-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.154431 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.154397 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-metrics-certs\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.155482 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.155449 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.177791 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.177754 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmv9\" (UniqueName: \"kubernetes.io/projected/4eac47eb-462c-49dd-8d15-cd6f90dac721-kube-api-access-kqmv9\") pod \"maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.206566 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.203783 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:47.512697 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.512657 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" event={"ID":"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45","Type":"ContainerStarted","Data":"245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c"} Oct 30 01:59:47.512890 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.512721 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" event={"ID":"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45","Type":"ContainerStarted","Data":"1a2cf063787f74644e976d23d05e6383e5833206da9d80dddaf9246ef9d937bc"} Oct 30 01:59:47.514663 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.514632 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" event={"ID":"90193f8e-9356-41c1-b6ba-8e465cdeead7","Type":"ContainerStarted","Data":"e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622"} Oct 30 01:59:47.514818 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.514687 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:47.546267 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.546228 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv"] Oct 30 01:59:47.548022 ip-10-0-142-110 kubenswrapper[5035]: W1030 01:59:47.547952 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eac47eb_462c_49dd_8d15_cd6f90dac721.slice/crio-57e11dee994d4924b083e8844e90eca9723ca9cf2c7910869b0b02233badf0be WatchSource:0}: Error finding container 57e11dee994d4924b083e8844e90eca9723ca9cf2c7910869b0b02233badf0be: Status 404 returned error can't find the container with id 57e11dee994d4924b083e8844e90eca9723ca9cf2c7910869b0b02233badf0be Oct 30 01:59:47.567728 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.566769 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" podStartSLOduration=1.5667455829999999 podStartE2EDuration="1.566745583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:47.562705375 +0000 UTC m=+1866.065637624" watchObservedRunningTime="2026-10-30 01:59:47.566745583 +0000 UTC m=+1866.069677804" Oct 30 01:59:47.619424 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:47.619262 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" podStartSLOduration=2.278133174 podStartE2EDuration="2.619233761s" totalImagesPullingTime="341.100587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:47.613984803 +0000 UTC m=+1866.116917024" watchObservedRunningTime="2026-10-30 01:59:47.619233761 +0000 UTC m=+1866.122165982" Oct 30 01:59:47.649436 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:47.649384 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 01:59:48.429231 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.429197 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:48.430847 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.430817 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:59:48.430974 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.430868 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:59:48.524987 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.524944 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" event={"ID":"4eac47eb-462c-49dd-8d15-cd6f90dac721","Type":"ContainerStarted","Data":"8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b"} Oct 30 01:59:48.525186 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.525015 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" event={"ID":"4eac47eb-462c-49dd-8d15-cd6f90dac721","Type":"ContainerStarted","Data":"57e11dee994d4924b083e8844e90eca9723ca9cf2c7910869b0b02233badf0be"} Oct 30 01:59:48.648604 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.648503 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" podStartSLOduration=2.285120886 podStartE2EDuration="2.648485061s" totalImagesPullingTime="363.364175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:48.588883967 +0000 UTC m=+1867.091816188" watchObservedRunningTime="2026-10-30 01:59:48.648485061 +0000 UTC m=+1867.151417373" Oct 30 01:59:48.649894 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.649867 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9"] Oct 30 01:59:48.650124 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.650101 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" containerID="cri-o://6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373" gracePeriod=30 Oct 30 01:59:48.653884 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.653856 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv"] Oct 30 01:59:48.654135 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:48.654101 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" containerID="cri-o://e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba" gracePeriod=30 Oct 30 01:59:49.429326 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:49.429274 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:59:49.429793 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:49.429349 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:59:49.532835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:49.532796 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:50.429252 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:50.429214 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:59:50.431522 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:50.429284 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:59:51.429432 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:51.429392 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:59:51.429670 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:51.429452 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:59:52.429358 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:52.429307 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:59:52.429857 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:52.429383 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:59:53.434851 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:53.434812 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:53.435861 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:53.435832 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:53.437174 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:53.437149 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 01:59:53.530658 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:53.530626 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 01:59:53.970422 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:53.970351 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:59:54.013578 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.013549 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:59:54.019594 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.019561 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-data\") pod \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " Oct 30 01:59:54.019749 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.019723 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-envoy\") pod \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " Oct 30 01:59:54.019928 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.019889 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-data" pod "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" (UID: "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.019994 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.019905 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-token\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-token\") pod \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " Oct 30 01:59:54.020038 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.020000 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-socket\") pod \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " Oct 30 01:59:54.020084 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.020067 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istiod-ca-cert\") pod \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " Oct 30 01:59:54.020139 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.020113 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-certs\") pod \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " Oct 30 01:59:54.020197 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.020146 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-credential-socket\") pod \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " Oct 30 01:59:54.020254 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.020194 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-podinfo\") pod \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " Oct 30 01:59:54.020313 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.020264 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-kube-api-access-mn87s\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-kube-api-access-mn87s\") pod \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\" (UID: \"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13\") " Oct 30 01:59:54.020846 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.020694 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-credential-socket" pod "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" (UID: "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.021033 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.021005 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-certs" pod "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" (UID: "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.021303 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.021283 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.021393 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.021313 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.021393 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.021330 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.021823 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.021793 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istiod-ca-cert" pod "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" (UID: "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:54.022243 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.022213 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-socket" pod "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" (UID: "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.022795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.022768 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-envoy" pod "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" (UID: "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.024016 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.023992 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-kube-api-access-mn87s" pod "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" (UID: "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13"). InnerVolumeSpecName "kube-api-access-mn87s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:54.024107 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.024010 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-podinfo" pod "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" (UID: "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:54.024384 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.024365 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-token" pod "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" (UID: "7b86e2b2-8b3d-4f5d-aea6-c19796c03c13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:54.122237 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122184 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-certs\") pod \"15c87830-ad96-4191-b4c3-22a6c537f46d\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " Oct 30 01:59:54.122423 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122328 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-socket\") pod \"15c87830-ad96-4191-b4c3-22a6c537f46d\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " Oct 30 01:59:54.122497 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122450 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-kube-api-access-bc2rc\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-kube-api-access-bc2rc\") pod \"15c87830-ad96-4191-b4c3-22a6c537f46d\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " Oct 30 01:59:54.122497 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122485 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-credential-socket\") pod \"15c87830-ad96-4191-b4c3-22a6c537f46d\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " Oct 30 01:59:54.122620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122513 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-data\") pod \"15c87830-ad96-4191-b4c3-22a6c537f46d\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " Oct 30 01:59:54.122620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122558 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-certs" pod "15c87830-ad96-4191-b4c3-22a6c537f46d" (UID: "15c87830-ad96-4191-b4c3-22a6c537f46d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.122620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122588 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-envoy\") pod \"15c87830-ad96-4191-b4c3-22a6c537f46d\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " Oct 30 01:59:54.122775 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122667 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-podinfo\") pod \"15c87830-ad96-4191-b4c3-22a6c537f46d\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " Oct 30 01:59:54.122830 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122772 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-token\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-token\") pod \"15c87830-ad96-4191-b4c3-22a6c537f46d\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " Oct 30 01:59:54.122830 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122784 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-socket" pod "15c87830-ad96-4191-b4c3-22a6c537f46d" (UID: "15c87830-ad96-4191-b4c3-22a6c537f46d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.122932 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122905 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c87830-ad96-4191-b4c3-22a6c537f46d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c87830-ad96-4191-b4c3-22a6c537f46d-istiod-ca-cert\") pod \"15c87830-ad96-4191-b4c3-22a6c537f46d\" (UID: \"15c87830-ad96-4191-b4c3-22a6c537f46d\") " Oct 30 01:59:54.122932 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.122911 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-credential-socket" pod "15c87830-ad96-4191-b4c3-22a6c537f46d" (UID: "15c87830-ad96-4191-b4c3-22a6c537f46d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.123143 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123118 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-data" pod "15c87830-ad96-4191-b4c3-22a6c537f46d" (UID: "15c87830-ad96-4191-b4c3-22a6c537f46d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.123598 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123573 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123603 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123620 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn87s\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-kube-api-access-mn87s\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123635 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123649 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123647 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c87830-ad96-4191-b4c3-22a6c537f46d-istiod-ca-cert" pod "15c87830-ad96-4191-b4c3-22a6c537f46d" (UID: "15c87830-ad96-4191-b4c3-22a6c537f46d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123662 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123676 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123689 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123701 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.123722 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.123715 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.125620 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.125587 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-kube-api-access-bc2rc" pod "15c87830-ad96-4191-b4c3-22a6c537f46d" (UID: "15c87830-ad96-4191-b4c3-22a6c537f46d"). InnerVolumeSpecName "kube-api-access-bc2rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:54.125835 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.125804 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-envoy" pod "15c87830-ad96-4191-b4c3-22a6c537f46d" (UID: "15c87830-ad96-4191-b4c3-22a6c537f46d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.126192 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.126164 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-token" pod "15c87830-ad96-4191-b4c3-22a6c537f46d" (UID: "15c87830-ad96-4191-b4c3-22a6c537f46d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:54.126671 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.126641 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-podinfo" pod "15c87830-ad96-4191-b4c3-22a6c537f46d" (UID: "15c87830-ad96-4191-b4c3-22a6c537f46d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:54.225795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.225700 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.225795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.225730 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.225795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.225741 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c87830-ad96-4191-b4c3-22a6c537f46d-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.225795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.225749 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc2rc\" (UniqueName: \"kubernetes.io/projected/15c87830-ad96-4191-b4c3-22a6c537f46d-kube-api-access-bc2rc\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.225795 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.225759 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c87830-ad96-4191-b4c3-22a6c537f46d-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.583429 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.583321 5035 generic.go:356] "Generic (PLEG): container finished" podID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerID="6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373" exitCode=0 Oct 30 01:59:54.583429 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.583390 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" Oct 30 01:59:54.583429 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.583407 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" event={"ID":"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13","Type":"ContainerDied","Data":"6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373"} Oct 30 01:59:54.584098 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.583470 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9" event={"ID":"7b86e2b2-8b3d-4f5d-aea6-c19796c03c13","Type":"ContainerDied","Data":"343b155b926e353a6c19e1ecdef2d8c0f0da41d0f1b8523decc97be0ffd64005"} Oct 30 01:59:54.584098 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.583494 5035 scope.go:122] "RemoveContainer" containerID="6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373" Oct 30 01:59:54.585783 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.585752 5035 generic.go:356] "Generic (PLEG): container finished" podID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerID="e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba" exitCode=0 Oct 30 01:59:54.586042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.585831 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" Oct 30 01:59:54.586042 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.585843 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" event={"ID":"15c87830-ad96-4191-b4c3-22a6c537f46d","Type":"ContainerDied","Data":"e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba"} Oct 30 01:59:54.586240 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.586207 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv" event={"ID":"15c87830-ad96-4191-b4c3-22a6c537f46d","Type":"ContainerDied","Data":"8e4e4574a464cfdea91b0e78c2329447c033ca597f02682decf540e64c82a6fe"} Oct 30 01:59:54.597921 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.597803 5035 scope.go:122] "RemoveContainer" containerID="6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373" Oct 30 01:59:54.598211 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:54.598188 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373\": container with ID starting with 6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373 not found: ID does not exist" containerID="6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373" Oct 30 01:59:54.598281 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.598219 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373"} err="failed to get container status \"6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373\": rpc error: code = NotFound desc = could not find container \"6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373\": container with ID starting with 6e914a8892a16f0b08bcae187c53d3be589f7447fd08e9f0d3a0c6eb6ed9d373 not found: ID does not exist" Oct 30 01:59:54.598281 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.598238 5035 scope.go:122] "RemoveContainer" containerID="e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba" Oct 30 01:59:54.608850 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.608820 5035 scope.go:122] "RemoveContainer" containerID="e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba" Oct 30 01:59:54.609183 ip-10-0-142-110 kubenswrapper[5035]: E1030 01:59:54.609162 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba\": container with ID starting with e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba not found: ID does not exist" containerID="e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba" Oct 30 01:59:54.609231 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.609195 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba"} err="failed to get container status \"e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba\": rpc error: code = NotFound desc = could not find container \"e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba\": container with ID starting with e5b06cce9d2db1ec18db8414f6a2a4987a838e7e70b5bc5e1d5510e99e9fb5ba not found: ID does not exist" Oct 30 01:59:54.643833 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.643786 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9"] Oct 30 01:59:54.656877 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.656838 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-dhtt9"] Oct 30 01:59:54.684876 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.684840 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv"] Oct 30 01:59:54.691632 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:54.691599 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-cqvsv"] Oct 30 01:59:55.546145 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:55.546107 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 01:59:55.647582 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:55.647517 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" path="/var/lib/kubelet/pods/15c87830-ad96-4191-b4c3-22a6c537f46d/volumes" Oct 30 01:59:55.648220 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:55.648191 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" path="/var/lib/kubelet/pods/7b86e2b2-8b3d-4f5d-aea6-c19796c03c13/volumes" Oct 30 01:59:56.181278 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:56.181242 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4"] Oct 30 01:59:56.181572 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:56.181504 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" containerID="cri-o://5e790a328ae4d9e08612cf0506a6a7e108236aef6ac4348eb715ab6a2310718a" gracePeriod=30 Oct 30 01:59:56.194778 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:56.194736 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx"] Oct 30 01:59:56.195049 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:56.194995 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" containerID="cri-o://73c1dd092aa52f0b9a477b1740d14614e55725bb78a14f355098109c57ad5ff3" gracePeriod=30 Oct 30 01:59:56.199017 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:56.198982 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld"] Oct 30 01:59:56.199273 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:56.199232 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" containerID="cri-o://a7e5bd42f6964847b701bcd96fe497a8cfb817f7789ad6787d6601f5048b41b5" gracePeriod=30 Oct 30 01:59:56.208691 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:56.208657 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7"] Oct 30 01:59:56.208937 ip-10-0-142-110 kubenswrapper[5035]: I1030 01:59:56.208908 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" containerID="cri-o://44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d" gracePeriod=30 Oct 30 02:00:00.171675 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.171627 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz"] Oct 30 02:00:00.172442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.172409 5035 cpu_manager.go:414] "containerMap: removing container" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" Oct 30 02:00:00.172442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.172438 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" Oct 30 02:00:00.172689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.172463 5035 cpu_manager.go:414] "containerMap: removing container" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" Oct 30 02:00:00.172689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.172470 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" Oct 30 02:00:00.172822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.172764 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c87830-ad96-4191-b4c3-22a6c537f46d" containerName="istio-proxy" Oct 30 02:00:00.172822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.172784 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b86e2b2-8b3d-4f5d-aea6-c19796c03c13" containerName="istio-proxy" Oct 30 02:00:00.179055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.179026 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:00.196943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.196900 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm"] Oct 30 02:00:00.203106 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.203075 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh"] Oct 30 02:00:00.206121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.203409 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:00.209344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.209313 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:00.222459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.222405 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz"] Oct 30 02:00:00.233704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.233665 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s"] Oct 30 02:00:00.241744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.241711 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:00.243070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.243041 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm"] Oct 30 02:00:00.259232 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.259181 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh"] Oct 30 02:00:00.273742 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.273709 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs"] Oct 30 02:00:00.278264 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.278232 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s"] Oct 30 02:00:00.278669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.278374 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:00.282046 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.282008 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs"] Oct 30 02:00:00.283930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.283903 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:00:00.324826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.324787 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d14078e-51dc-4bcf-9a9c-17ee7901be40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh\" (UID: \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:00.324826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.324832 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b822b327-db3b-459c-943a-39691f3875a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s\" (UID: \"b822b327-db3b-459c-943a-39691f3875a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:00.325103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.324942 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jc2z\" (UniqueName: \"kubernetes.io/projected/63e725b7-a078-45ca-aac5-1b0f5fcc177d-kube-api-access-5jc2z\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz\" (UID: \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:00.325103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.325047 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnxb\" (UniqueName: \"kubernetes.io/projected/b822b327-db3b-459c-943a-39691f3875a1-kube-api-access-xpnxb\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s\" (UID: \"b822b327-db3b-459c-943a-39691f3875a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:00.325103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.325080 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznnt\" (UniqueName: \"kubernetes.io/projected/0214eaff-b6f8-491f-9259-18431f48a390-kube-api-access-wznnt\") pod \"maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm\" (UID: \"0214eaff-b6f8-491f-9259-18431f48a390\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:00.325268 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.325230 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e725b7-a078-45ca-aac5-1b0f5fcc177d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz\" (UID: \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:00.325366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.325345 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98zl\" (UniqueName: \"kubernetes.io/projected/6d14078e-51dc-4bcf-9a9c-17ee7901be40-kube-api-access-p98zl\") pod \"maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh\" (UID: \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:00.325492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.325477 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0214eaff-b6f8-491f-9259-18431f48a390-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm\" (UID: \"0214eaff-b6f8-491f-9259-18431f48a390\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:00.392034 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.391997 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h"] Oct 30 02:00:00.397857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.397820 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.416495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.416296 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h"] Oct 30 02:00:00.416809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.416782 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-356082-openshift-default-dockercfg-84wdm\"" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.427717 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98zl\" (UniqueName: \"kubernetes.io/projected/6d14078e-51dc-4bcf-9a9c-17ee7901be40-kube-api-access-p98zl\") pod \"maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh\" (UID: \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.428768 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0214eaff-b6f8-491f-9259-18431f48a390-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm\" (UID: \"0214eaff-b6f8-491f-9259-18431f48a390\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.428884 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d14078e-51dc-4bcf-9a9c-17ee7901be40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh\" (UID: \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.428916 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b822b327-db3b-459c-943a-39691f3875a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s\" (UID: \"b822b327-db3b-459c-943a-39691f3875a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.428983 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkg5h\" (UniqueName: \"kubernetes.io/projected/14084516-d9fb-4b98-8609-3f55b46688e4-kube-api-access-nkg5h\") pod \"maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs\" (UID: \"14084516-d9fb-4b98-8609-3f55b46688e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.429015 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jc2z\" (UniqueName: \"kubernetes.io/projected/63e725b7-a078-45ca-aac5-1b0f5fcc177d-kube-api-access-5jc2z\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz\" (UID: \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.429050 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14084516-d9fb-4b98-8609-3f55b46688e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs\" (UID: \"14084516-d9fb-4b98-8609-3f55b46688e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.429105 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnxb\" (UniqueName: \"kubernetes.io/projected/b822b327-db3b-459c-943a-39691f3875a1-kube-api-access-xpnxb\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s\" (UID: \"b822b327-db3b-459c-943a-39691f3875a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.429149 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznnt\" (UniqueName: \"kubernetes.io/projected/0214eaff-b6f8-491f-9259-18431f48a390-kube-api-access-wznnt\") pod \"maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm\" (UID: \"0214eaff-b6f8-491f-9259-18431f48a390\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.429264 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e725b7-a078-45ca-aac5-1b0f5fcc177d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz\" (UID: \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.430120 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e725b7-a078-45ca-aac5-1b0f5fcc177d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz\" (UID: \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.430282 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0214eaff-b6f8-491f-9259-18431f48a390-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm\" (UID: \"0214eaff-b6f8-491f-9259-18431f48a390\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.430767 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d14078e-51dc-4bcf-9a9c-17ee7901be40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh\" (UID: \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:00.431640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.431298 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b822b327-db3b-459c-943a-39691f3875a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s\" (UID: \"b822b327-db3b-459c-943a-39691f3875a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:00.456226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.456186 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98zl\" (UniqueName: \"kubernetes.io/projected/6d14078e-51dc-4bcf-9a9c-17ee7901be40-kube-api-access-p98zl\") pod \"maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh\" (UID: \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:00.460065 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.460029 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jc2z\" (UniqueName: \"kubernetes.io/projected/63e725b7-a078-45ca-aac5-1b0f5fcc177d-kube-api-access-5jc2z\") pod \"maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz\" (UID: \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:00.460718 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.460685 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznnt\" (UniqueName: \"kubernetes.io/projected/0214eaff-b6f8-491f-9259-18431f48a390-kube-api-access-wznnt\") pod \"maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm\" (UID: \"0214eaff-b6f8-491f-9259-18431f48a390\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:00.460865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.460722 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnxb\" (UniqueName: \"kubernetes.io/projected/b822b327-db3b-459c-943a-39691f3875a1-kube-api-access-xpnxb\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s\" (UID: \"b822b327-db3b-459c-943a-39691f3875a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:00.496789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.496748 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:00.530803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.530770 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-data\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.530982 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.530824 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879878f-0211-4689-b111-62aead00bccf-istio-podinfo\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.531049 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.530976 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkg5h\" (UniqueName: \"kubernetes.io/projected/14084516-d9fb-4b98-8609-3f55b46688e4-kube-api-access-nkg5h\") pod \"maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs\" (UID: \"14084516-d9fb-4b98-8609-3f55b46688e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:00.531100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.531044 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14084516-d9fb-4b98-8609-3f55b46688e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs\" (UID: \"14084516-d9fb-4b98-8609-3f55b46688e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:00.531100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.531073 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-istio-token\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.531272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.531242 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgkr\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-kube-api-access-drgkr\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.531359 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.531341 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-envoy\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.531511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.531489 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-socket\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.531602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.531556 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-credential-socket\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.531651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.531634 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879878f-0211-4689-b111-62aead00bccf-istiod-ca-cert\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.531715 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.531702 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-certs\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.531769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.531752 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14084516-d9fb-4b98-8609-3f55b46688e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs\" (UID: \"14084516-d9fb-4b98-8609-3f55b46688e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:00.541346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.541310 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:00.569233 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.569174 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:00.576681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.576291 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:00.579417 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.578975 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkg5h\" (UniqueName: \"kubernetes.io/projected/14084516-d9fb-4b98-8609-3f55b46688e4-kube-api-access-nkg5h\") pod \"maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs\" (UID: \"14084516-d9fb-4b98-8609-3f55b46688e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:00.669555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.648859 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:00.699839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.698758 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgkr\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-kube-api-access-drgkr\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.699839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.698841 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-envoy\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.699839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.698924 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-socket\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.699839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.698964 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-credential-socket\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.699839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.698996 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879878f-0211-4689-b111-62aead00bccf-istiod-ca-cert\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.699839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.699053 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-certs\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.699839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.699175 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-data\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.699839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.699219 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879878f-0211-4689-b111-62aead00bccf-istio-podinfo\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.699839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.699267 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-istio-token\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.746594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.702144 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-data\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.746594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.702367 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-certs\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.746594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.702584 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-socket\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.746594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.702784 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-credential-socket\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.752448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.752416 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879878f-0211-4689-b111-62aead00bccf-istiod-ca-cert\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.753975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.753938 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-istio-token\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.754648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.754611 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgkr\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-kube-api-access-drgkr\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.754832 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.754756 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879878f-0211-4689-b111-62aead00bccf-istio-podinfo\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:00.778175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:00.778143 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-envoy\") pod \"e2e-api-b-356082-openshift-default-7dd6879f-cf82h\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:01.036638 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:01.036576 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e725b7_a078_45ca_aac5_1b0f5fcc177d.slice/crio-b7098f200ead09cd9a9c12a4861df07cf410ec3cdb70071f252ae1dac3072a4c WatchSource:0}: Error finding container b7098f200ead09cd9a9c12a4861df07cf410ec3cdb70071f252ae1dac3072a4c: Status 404 returned error can't find the container with id b7098f200ead09cd9a9c12a4861df07cf410ec3cdb70071f252ae1dac3072a4c Oct 30 02:00:01.043294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.043247 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz"] Oct 30 02:00:01.046510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.046431 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:01.105675 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.105623 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm"] Oct 30 02:00:01.119998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.119961 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh"] Oct 30 02:00:01.132122 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:01.131710 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d14078e_51dc_4bcf_9a9c_17ee7901be40.slice/crio-e01a4a008989fe77f1519b1a66cab4cf047e50bc00fb0fef650ddcffa3d77aa4 WatchSource:0}: Error finding container e01a4a008989fe77f1519b1a66cab4cf047e50bc00fb0fef650ddcffa3d77aa4: Status 404 returned error can't find the container with id e01a4a008989fe77f1519b1a66cab4cf047e50bc00fb0fef650ddcffa3d77aa4 Oct 30 02:00:01.147840 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.147804 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs"] Oct 30 02:00:01.159973 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.152745 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s"] Oct 30 02:00:01.168029 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:01.167996 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb822b327_db3b_459c_943a_39691f3875a1.slice/crio-e9264d47280a956a41866381f2ae9b251e8fc2c65e0fd6ed23a8106d2b700613 WatchSource:0}: Error finding container e9264d47280a956a41866381f2ae9b251e8fc2c65e0fd6ed23a8106d2b700613: Status 404 returned error can't find the container with id e9264d47280a956a41866381f2ae9b251e8fc2c65e0fd6ed23a8106d2b700613 Oct 30 02:00:01.382248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.381651 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h"] Oct 30 02:00:01.421588 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:01.419395 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3879878f_0211_4689_b111_62aead00bccf.slice/crio-e1a0d4a4eb94645d5c918b88026d0de3eecbd7a51627ef9248f876868e305f17 WatchSource:0}: Error finding container e1a0d4a4eb94645d5c918b88026d0de3eecbd7a51627ef9248f876868e305f17: Status 404 returned error can't find the container with id e1a0d4a4eb94645d5c918b88026d0de3eecbd7a51627ef9248f876868e305f17 Oct 30 02:00:01.436113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.433228 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:01.436113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.433330 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:01.436113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.433371 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:01.798087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.798052 5035 generic.go:356] "Generic (PLEG): container finished" podID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerID="73c1dd092aa52f0b9a477b1740d14614e55725bb78a14f355098109c57ad5ff3" exitCode=0 Oct 30 02:00:01.798254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.798120 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" event={"ID":"5c9e1898-ff05-4c0c-bf18-858d74c2b636","Type":"ContainerDied","Data":"73c1dd092aa52f0b9a477b1740d14614e55725bb78a14f355098109c57ad5ff3"} Oct 30 02:00:01.798254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.798181 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" event={"ID":"5c9e1898-ff05-4c0c-bf18-858d74c2b636","Type":"ContainerDied","Data":"622c350f9a16a665dbece40e6aca568cd3e94f43dac246f6c17e0763e5e3d65f"} Oct 30 02:00:01.798254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.798211 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="622c350f9a16a665dbece40e6aca568cd3e94f43dac246f6c17e0763e5e3d65f" Oct 30 02:00:01.799671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.799647 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 02:00:01.799825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.799719 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" event={"ID":"b822b327-db3b-459c-943a-39691f3875a1","Type":"ContainerStarted","Data":"e9264d47280a956a41866381f2ae9b251e8fc2c65e0fd6ed23a8106d2b700613"} Oct 30 02:00:01.801468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.801404 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" event={"ID":"63e725b7-a078-45ca-aac5-1b0f5fcc177d","Type":"ContainerStarted","Data":"b7098f200ead09cd9a9c12a4861df07cf410ec3cdb70071f252ae1dac3072a4c"} Oct 30 02:00:01.804141 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.804108 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" event={"ID":"14084516-d9fb-4b98-8609-3f55b46688e4","Type":"ContainerStarted","Data":"a8d6b501eb23b9f7c3395e058b073fccebc38d8a495003f83735bdb35c99206b"} Oct 30 02:00:01.806605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.806509 5035 generic.go:356] "Generic (PLEG): container finished" podID="d5083785-8e42-4372-8d43-49cc6edceda4" containerID="44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d" exitCode=0 Oct 30 02:00:01.806740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.806606 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" Oct 30 02:00:01.806740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.806646 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" event={"ID":"d5083785-8e42-4372-8d43-49cc6edceda4","Type":"ContainerDied","Data":"44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d"} Oct 30 02:00:01.806740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.806689 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7" event={"ID":"d5083785-8e42-4372-8d43-49cc6edceda4","Type":"ContainerDied","Data":"c5c1b9c1e3fa31fd259ae90894fc6242dc753b4c8ba1ac0e4ae6c46053313612"} Oct 30 02:00:01.806921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.806721 5035 scope.go:122] "RemoveContainer" containerID="44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d" Oct 30 02:00:01.808770 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.808742 5035 generic.go:356] "Generic (PLEG): container finished" podID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerID="a7e5bd42f6964847b701bcd96fe497a8cfb817f7789ad6787d6601f5048b41b5" exitCode=0 Oct 30 02:00:01.808995 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.808785 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" event={"ID":"d07e40cb-209d-47ee-9324-cd445c485cb6","Type":"ContainerDied","Data":"a7e5bd42f6964847b701bcd96fe497a8cfb817f7789ad6787d6601f5048b41b5"} Oct 30 02:00:01.809086 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.809041 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" event={"ID":"d07e40cb-209d-47ee-9324-cd445c485cb6","Type":"ContainerDied","Data":"5a8778230750b6e15c24d896c866c2cceb3e707e68bfb09f5ac4c81956810534"} Oct 30 02:00:01.809086 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.809070 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a8778230750b6e15c24d896c866c2cceb3e707e68bfb09f5ac4c81956810534" Oct 30 02:00:01.810869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.810846 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" event={"ID":"6d14078e-51dc-4bcf-9a9c-17ee7901be40","Type":"ContainerStarted","Data":"e01a4a008989fe77f1519b1a66cab4cf047e50bc00fb0fef650ddcffa3d77aa4"} Oct 30 02:00:01.826305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.824080 5035 generic.go:356] "Generic (PLEG): container finished" podID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerID="5e790a328ae4d9e08612cf0506a6a7e108236aef6ac4348eb715ab6a2310718a" exitCode=0 Oct 30 02:00:01.826305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.824211 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" event={"ID":"a5ce2c95-167f-40f3-9747-f411d0bc8ea5","Type":"ContainerDied","Data":"5e790a328ae4d9e08612cf0506a6a7e108236aef6ac4348eb715ab6a2310718a"} Oct 30 02:00:01.826305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.824265 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" event={"ID":"a5ce2c95-167f-40f3-9747-f411d0bc8ea5","Type":"ContainerDied","Data":"ef395140ebde85c79b41eb1856eeb6335d8738b4d38ff3e46db4a1b6092ac943"} Oct 30 02:00:01.826305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.824288 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef395140ebde85c79b41eb1856eeb6335d8738b4d38ff3e46db4a1b6092ac943" Oct 30 02:00:01.830229 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.827674 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" event={"ID":"3879878f-0211-4689-b111-62aead00bccf","Type":"ContainerStarted","Data":"e1a0d4a4eb94645d5c918b88026d0de3eecbd7a51627ef9248f876868e305f17"} Oct 30 02:00:01.834060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.834011 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" event={"ID":"0214eaff-b6f8-491f-9259-18431f48a390","Type":"ContainerStarted","Data":"208bd414a900d709ae32c787908983a56ead8a8a0a23adf489a4246a5650a361"} Oct 30 02:00:01.834180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.834075 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" event={"ID":"0214eaff-b6f8-491f-9259-18431f48a390","Type":"ContainerStarted","Data":"7895686aeab107be27e5e58bd5f0d09fcb7ccf4766359fef98ada90bb3bb1083"} Oct 30 02:00:01.840888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.840854 5035 scope.go:122] "RemoveContainer" containerID="44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d" Oct 30 02:00:01.841291 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:01.841233 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d\": container with ID starting with 44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d not found: ID does not exist" containerID="44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d" Oct 30 02:00:01.841426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.841296 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d"} err="failed to get container status \"44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d\": rpc error: code = NotFound desc = could not find container \"44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d\": container with ID starting with 44ad78d405151d5dc0de98ddcdedd5c69d6e15e229ae8753e702fb57d2f5483d not found: ID does not exist" Oct 30 02:00:01.841496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.841470 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 02:00:01.844852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.844774 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-data\") pod \"d07e40cb-209d-47ee-9324-cd445c485cb6\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " Oct 30 02:00:01.844852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.844834 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-envoy\") pod \"d07e40cb-209d-47ee-9324-cd445c485cb6\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " Oct 30 02:00:01.845004 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.844864 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-socket\") pod \"d07e40cb-209d-47ee-9324-cd445c485cb6\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " Oct 30 02:00:01.845004 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.844915 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-podinfo\") pod \"d07e40cb-209d-47ee-9324-cd445c485cb6\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " Oct 30 02:00:01.845004 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.844944 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-istio-token\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-istio-token\") pod \"d5083785-8e42-4372-8d43-49cc6edceda4\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " Oct 30 02:00:01.845004 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.844973 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-token\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-token\") pod \"d07e40cb-209d-47ee-9324-cd445c485cb6\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " Oct 30 02:00:01.845164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.845036 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-kube-api-access-k7lcg\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-kube-api-access-k7lcg\") pod \"d07e40cb-209d-47ee-9324-cd445c485cb6\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " Oct 30 02:00:01.845164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.845065 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-data" pod "d07e40cb-209d-47ee-9324-cd445c485cb6" (UID: "d07e40cb-209d-47ee-9324-cd445c485cb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.845272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.845252 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-socket\") pod \"d5083785-8e42-4372-8d43-49cc6edceda4\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " Oct 30 02:00:01.845383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.845365 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-certs\") pod \"d07e40cb-209d-47ee-9324-cd445c485cb6\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " Oct 30 02:00:01.845606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.845503 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-credential-socket\") pod \"d07e40cb-209d-47ee-9324-cd445c485cb6\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " Oct 30 02:00:01.846408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.846386 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5083785-8e42-4372-8d43-49cc6edceda4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5083785-8e42-4372-8d43-49cc6edceda4-istio-podinfo\") pod \"d5083785-8e42-4372-8d43-49cc6edceda4\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " Oct 30 02:00:01.846634 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.846615 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-certs\") pod \"d5083785-8e42-4372-8d43-49cc6edceda4\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " Oct 30 02:00:01.846781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.846764 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-kube-api-access-lhhg4\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-kube-api-access-lhhg4\") pod \"d5083785-8e42-4372-8d43-49cc6edceda4\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " Oct 30 02:00:01.846971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.846949 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07e40cb-209d-47ee-9324-cd445c485cb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07e40cb-209d-47ee-9324-cd445c485cb6-istiod-ca-cert\") pod \"d07e40cb-209d-47ee-9324-cd445c485cb6\" (UID: \"d07e40cb-209d-47ee-9324-cd445c485cb6\") " Oct 30 02:00:01.847110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.847095 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-envoy\") pod \"d5083785-8e42-4372-8d43-49cc6edceda4\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " Oct 30 02:00:01.847237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.847222 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-credential-socket\") pod \"d5083785-8e42-4372-8d43-49cc6edceda4\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " Oct 30 02:00:01.847409 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.847392 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5083785-8e42-4372-8d43-49cc6edceda4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5083785-8e42-4372-8d43-49cc6edceda4-istiod-ca-cert\") pod \"d5083785-8e42-4372-8d43-49cc6edceda4\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " Oct 30 02:00:01.847676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.847657 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-data\") pod \"d5083785-8e42-4372-8d43-49cc6edceda4\" (UID: \"d5083785-8e42-4372-8d43-49cc6edceda4\") " Oct 30 02:00:01.848519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.846517 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-certs" pod "d07e40cb-209d-47ee-9324-cd445c485cb6" (UID: "d07e40cb-209d-47ee-9324-cd445c485cb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.848632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.846814 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-credential-socket" pod "d07e40cb-209d-47ee-9324-cd445c485cb6" (UID: "d07e40cb-209d-47ee-9324-cd445c485cb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.848632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.848306 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-socket" pod "d07e40cb-209d-47ee-9324-cd445c485cb6" (UID: "d07e40cb-209d-47ee-9324-cd445c485cb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.848759 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.848692 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-certs" pod "d5083785-8e42-4372-8d43-49cc6edceda4" (UID: "d5083785-8e42-4372-8d43-49cc6edceda4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.849393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849101 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07e40cb-209d-47ee-9324-cd445c485cb6-istiod-ca-cert" pod "d07e40cb-209d-47ee-9324-cd445c485cb6" (UID: "d07e40cb-209d-47ee-9324-cd445c485cb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:01.849393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849158 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-credential-socket" pod "d5083785-8e42-4372-8d43-49cc6edceda4" (UID: "d5083785-8e42-4372-8d43-49cc6edceda4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.849551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849453 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-socket" pod "d5083785-8e42-4372-8d43-49cc6edceda4" (UID: "d5083785-8e42-4372-8d43-49cc6edceda4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.849620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849589 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-data" pod "d5083785-8e42-4372-8d43-49cc6edceda4" (UID: "d5083785-8e42-4372-8d43-49cc6edceda4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.849675 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849630 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5083785-8e42-4372-8d43-49cc6edceda4-istiod-ca-cert" pod "d5083785-8e42-4372-8d43-49cc6edceda4" (UID: "d5083785-8e42-4372-8d43-49cc6edceda4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:01.849773 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849745 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.849898 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849778 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.849898 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849796 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.849898 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849810 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.849898 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849823 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.849898 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849838 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07e40cb-209d-47ee-9324-cd445c485cb6-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.849898 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.849852 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.851835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.851808 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-kube-api-access-lhhg4" pod "d5083785-8e42-4372-8d43-49cc6edceda4" (UID: "d5083785-8e42-4372-8d43-49cc6edceda4"). InnerVolumeSpecName "kube-api-access-lhhg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:01.851961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.851852 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-istio-token" pod "d5083785-8e42-4372-8d43-49cc6edceda4" (UID: "d5083785-8e42-4372-8d43-49cc6edceda4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:01.851961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.851909 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-token" pod "d07e40cb-209d-47ee-9324-cd445c485cb6" (UID: "d07e40cb-209d-47ee-9324-cd445c485cb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:01.851961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.851919 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-envoy" pod "d5083785-8e42-4372-8d43-49cc6edceda4" (UID: "d5083785-8e42-4372-8d43-49cc6edceda4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.852105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.851980 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-kube-api-access-k7lcg" pod "d07e40cb-209d-47ee-9324-cd445c485cb6" (UID: "d07e40cb-209d-47ee-9324-cd445c485cb6"). InnerVolumeSpecName "kube-api-access-k7lcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:01.852105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.852001 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-podinfo" pod "d07e40cb-209d-47ee-9324-cd445c485cb6" (UID: "d07e40cb-209d-47ee-9324-cd445c485cb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:00:01.852105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.852030 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-envoy" pod "d07e40cb-209d-47ee-9324-cd445c485cb6" (UID: "d07e40cb-209d-47ee-9324-cd445c485cb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.854994 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.854967 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5083785-8e42-4372-8d43-49cc6edceda4-istio-podinfo" pod "d5083785-8e42-4372-8d43-49cc6edceda4" (UID: "d5083785-8e42-4372-8d43-49cc6edceda4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:00:01.888506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.888473 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 02:00:01.894348 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.894320 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 02:00:01.911732 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.911660 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" podStartSLOduration=1.9116354169999998 podStartE2EDuration="1.911635417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:01.905343604 +0000 UTC m=+1880.408275835" watchObservedRunningTime="2026-10-30 02:00:01.911635417 +0000 UTC m=+1880.414567638" Oct 30 02:00:01.951073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951037 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istiod-ca-cert\") pod \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " Oct 30 02:00:01.951277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951102 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-socket\") pod \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " Oct 30 02:00:01.951277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951156 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-socket\") pod \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " Oct 30 02:00:01.951277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951219 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-data\") pod \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " Oct 30 02:00:01.951277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951273 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-token\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-token\") pod \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " Oct 30 02:00:01.951487 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951317 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-envoy\") pod \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " Oct 30 02:00:01.951487 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951405 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-certs\") pod \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " Oct 30 02:00:01.951487 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951426 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-socket" pod "a5ce2c95-167f-40f3-9747-f411d0bc8ea5" (UID: "a5ce2c95-167f-40f3-9747-f411d0bc8ea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.951671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951577 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-socket" pod "5c9e1898-ff05-4c0c-bf18-858d74c2b636" (UID: "5c9e1898-ff05-4c0c-bf18-858d74c2b636"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.951671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951439 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-kube-api-access-dkfx8\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-kube-api-access-dkfx8\") pod \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " Oct 30 02:00:01.951671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951617 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-data" pod "5c9e1898-ff05-4c0c-bf18-858d74c2b636" (UID: "5c9e1898-ff05-4c0c-bf18-858d74c2b636"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.951671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951639 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istiod-ca-cert" pod "a5ce2c95-167f-40f3-9747-f411d0bc8ea5" (UID: "a5ce2c95-167f-40f3-9747-f411d0bc8ea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:01.951928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951680 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-podinfo\") pod \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " Oct 30 02:00:01.951928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951826 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-certs" pod "a5ce2c95-167f-40f3-9747-f411d0bc8ea5" (UID: "a5ce2c95-167f-40f3-9747-f411d0bc8ea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.951928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951847 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-envoy\") pod \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " Oct 30 02:00:01.951928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951892 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-token\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-token\") pod \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " Oct 30 02:00:01.952128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951954 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istiod-ca-cert\") pod \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " Oct 30 02:00:01.952128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.951992 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-data\") pod \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " Oct 30 02:00:01.952128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.952053 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-kube-api-access-5j74g\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-kube-api-access-5j74g\") pod \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " Oct 30 02:00:01.952128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.952084 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-credential-socket\") pod \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " Oct 30 02:00:01.952128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.952117 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-podinfo\") pod \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " Oct 30 02:00:01.952384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.952179 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-credential-socket\") pod \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\" (UID: \"a5ce2c95-167f-40f3-9747-f411d0bc8ea5\") " Oct 30 02:00:01.952384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.952224 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-certs\") pod \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\" (UID: \"5c9e1898-ff05-4c0c-bf18-858d74c2b636\") " Oct 30 02:00:01.952384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.952364 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-data" pod "a5ce2c95-167f-40f3-9747-f411d0bc8ea5" (UID: "a5ce2c95-167f-40f3-9747-f411d0bc8ea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.952965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.952707 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istiod-ca-cert" pod "5c9e1898-ff05-4c0c-bf18-858d74c2b636" (UID: "5c9e1898-ff05-4c0c-bf18-858d74c2b636"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:01.952965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.952934 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-certs" pod "5c9e1898-ff05-4c0c-bf18-858d74c2b636" (UID: "5c9e1898-ff05-4c0c-bf18-858d74c2b636"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953345 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953375 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953389 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953402 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953415 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953428 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7lcg\" (UniqueName: \"kubernetes.io/projected/d07e40cb-209d-47ee-9324-cd445c485cb6-kube-api-access-k7lcg\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953446 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953459 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953472 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953485 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953498 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5083785-8e42-4372-8d43-49cc6edceda4-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953549 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhhg4\" (UniqueName: \"kubernetes.io/projected/d5083785-8e42-4372-8d43-49cc6edceda4-kube-api-access-lhhg4\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953565 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953578 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953592 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953608 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5083785-8e42-4372-8d43-49cc6edceda4-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953622 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953635 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.953940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.953649 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5083785-8e42-4372-8d43-49cc6edceda4-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:01.955925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.955885 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-credential-socket" pod "5c9e1898-ff05-4c0c-bf18-858d74c2b636" (UID: "5c9e1898-ff05-4c0c-bf18-858d74c2b636"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:01.956043 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:01.955953 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-credential-socket" pod "a5ce2c95-167f-40f3-9747-f411d0bc8ea5" (UID: "a5ce2c95-167f-40f3-9747-f411d0bc8ea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:02.015068 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.014003 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-envoy" pod "5c9e1898-ff05-4c0c-bf18-858d74c2b636" (UID: "5c9e1898-ff05-4c0c-bf18-858d74c2b636"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:02.015581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.015551 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-envoy" pod "a5ce2c95-167f-40f3-9747-f411d0bc8ea5" (UID: "a5ce2c95-167f-40f3-9747-f411d0bc8ea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:02.016105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.016070 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-kube-api-access-dkfx8" pod "a5ce2c95-167f-40f3-9747-f411d0bc8ea5" (UID: "a5ce2c95-167f-40f3-9747-f411d0bc8ea5"). InnerVolumeSpecName "kube-api-access-dkfx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:02.016700 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.016660 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-token" pod "5c9e1898-ff05-4c0c-bf18-858d74c2b636" (UID: "5c9e1898-ff05-4c0c-bf18-858d74c2b636"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:02.017085 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.017058 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-podinfo" pod "a5ce2c95-167f-40f3-9747-f411d0bc8ea5" (UID: "a5ce2c95-167f-40f3-9747-f411d0bc8ea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:00:02.017467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.017440 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-kube-api-access-5j74g" pod "5c9e1898-ff05-4c0c-bf18-858d74c2b636" (UID: "5c9e1898-ff05-4c0c-bf18-858d74c2b636"). InnerVolumeSpecName "kube-api-access-5j74g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:02.017969 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.017945 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-podinfo" pod "5c9e1898-ff05-4c0c-bf18-858d74c2b636" (UID: "5c9e1898-ff05-4c0c-bf18-858d74c2b636"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:00:02.018170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.018149 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-token" pod "a5ce2c95-167f-40f3-9747-f411d0bc8ea5" (UID: "a5ce2c95-167f-40f3-9747-f411d0bc8ea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056361 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j74g\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-kube-api-access-5j74g\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056400 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056414 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056428 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056441 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056454 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9e1898-ff05-4c0c-bf18-858d74c2b636-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056466 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkfx8\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-kube-api-access-dkfx8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056480 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056493 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.056577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.056507 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ce2c95-167f-40f3-9747-f411d0bc8ea5-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:02.163741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.163704 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7"] Oct 30 02:00:02.170118 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.170084 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-tfvp7"] Oct 30 02:00:02.843835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.843727 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" event={"ID":"b822b327-db3b-459c-943a-39691f3875a1","Type":"ContainerStarted","Data":"63b8bdbb1046267fdc392618cb49714ca5aaa9a5e4e53d7b7c9c7fd43c1bfbb6"} Oct 30 02:00:02.845873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.845842 5035 generic.go:356] "Generic (PLEG): container finished" podID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerID="bcecdd6208bf5145b929b0f7fe832796c26589ff3014f62814b030bdc97a6889" exitCode=60 Oct 30 02:00:02.846055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.845930 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" event={"ID":"63e725b7-a078-45ca-aac5-1b0f5fcc177d","Type":"ContainerDied","Data":"bcecdd6208bf5145b929b0f7fe832796c26589ff3014f62814b030bdc97a6889"} Oct 30 02:00:02.846254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.846238 5035 scope.go:122] "RemoveContainer" containerID="bcecdd6208bf5145b929b0f7fe832796c26589ff3014f62814b030bdc97a6889" Oct 30 02:00:02.847939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.847915 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" event={"ID":"14084516-d9fb-4b98-8609-3f55b46688e4","Type":"ContainerStarted","Data":"c624369ae042c591e2bf5d77680eea4a919cbc8fd7cb4fda37649b081f07925d"} Oct 30 02:00:02.850669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.850648 5035 generic.go:356] "Generic (PLEG): container finished" podID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerID="4a167a589f69b3ccfb888d1f69be43b9a2d08a55e8da6495265da36dde8922ce" exitCode=60 Oct 30 02:00:02.850776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.850734 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" event={"ID":"6d14078e-51dc-4bcf-9a9c-17ee7901be40","Type":"ContainerDied","Data":"4a167a589f69b3ccfb888d1f69be43b9a2d08a55e8da6495265da36dde8922ce"} Oct 30 02:00:02.851037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.851018 5035 scope.go:122] "RemoveContainer" containerID="4a167a589f69b3ccfb888d1f69be43b9a2d08a55e8da6495265da36dde8922ce" Oct 30 02:00:02.852455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.852431 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" event={"ID":"3879878f-0211-4689-b111-62aead00bccf","Type":"ContainerStarted","Data":"cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684"} Oct 30 02:00:02.854125 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.854098 5035 generic.go:356] "Generic (PLEG): container finished" podID="0214eaff-b6f8-491f-9259-18431f48a390" containerID="208bd414a900d709ae32c787908983a56ead8a8a0a23adf489a4246a5650a361" exitCode=60 Oct 30 02:00:02.854237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.854172 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" event={"ID":"0214eaff-b6f8-491f-9259-18431f48a390","Type":"ContainerDied","Data":"208bd414a900d709ae32c787908983a56ead8a8a0a23adf489a4246a5650a361"} Oct 30 02:00:02.854237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.854220 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld" Oct 30 02:00:02.854405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.854384 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx" Oct 30 02:00:02.854452 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.854406 5035 scope.go:122] "RemoveContainer" containerID="208bd414a900d709ae32c787908983a56ead8a8a0a23adf489a4246a5650a361" Oct 30 02:00:02.854540 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.854515 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4" Oct 30 02:00:02.940599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.940539 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" podStartSLOduration=2.940505591 podStartE2EDuration="2.940505591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:02.89485764 +0000 UTC m=+1881.397789861" watchObservedRunningTime="2026-10-30 02:00:02.940505591 +0000 UTC m=+1881.443437811" Oct 30 02:00:02.943629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:02.943578 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" podStartSLOduration=2.943563154 podStartE2EDuration="2.943563154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:02.939627665 +0000 UTC m=+1881.442559886" watchObservedRunningTime="2026-10-30 02:00:02.943563154 +0000 UTC m=+1881.446495373" Oct 30 02:00:03.054076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.052756 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:03.104602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.103357 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:03.183600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.183523 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" podStartSLOduration=3.183501179 podStartE2EDuration="3.183501179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.179849956 +0000 UTC m=+1881.682782179" watchObservedRunningTime="2026-10-30 02:00:03.183501179 +0000 UTC m=+1881.686433405" Oct 30 02:00:03.237140 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.237101 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld"] Oct 30 02:00:03.243258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.243221 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-q68ld"] Oct 30 02:00:03.279827 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.279799 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx"] Oct 30 02:00:03.287445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.287411 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-jl2gx"] Oct 30 02:00:03.457930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.456665 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4"] Oct 30 02:00:03.476990 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.476953 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-pzjz4"] Oct 30 02:00:03.645415 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.645375 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" path="/var/lib/kubelet/pods/5c9e1898-ff05-4c0c-bf18-858d74c2b636/volumes" Oct 30 02:00:03.646047 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.646020 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" path="/var/lib/kubelet/pods/a5ce2c95-167f-40f3-9747-f411d0bc8ea5/volumes" Oct 30 02:00:03.646553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.646521 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" path="/var/lib/kubelet/pods/d07e40cb-209d-47ee-9324-cd445c485cb6/volumes" Oct 30 02:00:03.647374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.647358 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" path="/var/lib/kubelet/pods/d5083785-8e42-4372-8d43-49cc6edceda4/volumes" Oct 30 02:00:03.865092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.864994 5035 generic.go:356] "Generic (PLEG): container finished" podID="b822b327-db3b-459c-943a-39691f3875a1" containerID="63b8bdbb1046267fdc392618cb49714ca5aaa9a5e4e53d7b7c9c7fd43c1bfbb6" exitCode=60 Oct 30 02:00:03.865577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.865079 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" event={"ID":"b822b327-db3b-459c-943a-39691f3875a1","Type":"ContainerDied","Data":"63b8bdbb1046267fdc392618cb49714ca5aaa9a5e4e53d7b7c9c7fd43c1bfbb6"} Oct 30 02:00:03.865577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.865408 5035 scope.go:122] "RemoveContainer" containerID="63b8bdbb1046267fdc392618cb49714ca5aaa9a5e4e53d7b7c9c7fd43c1bfbb6" Oct 30 02:00:03.867310 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.867274 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" event={"ID":"63e725b7-a078-45ca-aac5-1b0f5fcc177d","Type":"ContainerStarted","Data":"cc2b8c9d2f4d3bcfa0483e6a9e340eac9133201e9e49fc1bcedeb4ac48405bdc"} Oct 30 02:00:03.870156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.869664 5035 generic.go:356] "Generic (PLEG): container finished" podID="14084516-d9fb-4b98-8609-3f55b46688e4" containerID="c624369ae042c591e2bf5d77680eea4a919cbc8fd7cb4fda37649b081f07925d" exitCode=60 Oct 30 02:00:03.870156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.869814 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" event={"ID":"14084516-d9fb-4b98-8609-3f55b46688e4","Type":"ContainerDied","Data":"c624369ae042c591e2bf5d77680eea4a919cbc8fd7cb4fda37649b081f07925d"} Oct 30 02:00:03.870453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.870431 5035 scope.go:122] "RemoveContainer" containerID="c624369ae042c591e2bf5d77680eea4a919cbc8fd7cb4fda37649b081f07925d" Oct 30 02:00:03.872802 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.872773 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" event={"ID":"6d14078e-51dc-4bcf-9a9c-17ee7901be40","Type":"ContainerStarted","Data":"fc16297508a87b39cd9b53913b5d6d8cea8daf44e40b78ac236dec55c737f477"} Oct 30 02:00:03.875295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.875265 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" event={"ID":"0214eaff-b6f8-491f-9259-18431f48a390","Type":"ContainerStarted","Data":"7fa054c9988d1c688488e2844e1bdee1a4741f3f21896509539a184961801810"} Oct 30 02:00:03.875619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.875593 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:03.876772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.876751 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:00:03.995517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:03.995458 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" podStartSLOduration=3.995442828 podStartE2EDuration="3.995442828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.994844321 +0000 UTC m=+1882.497776542" watchObservedRunningTime="2026-10-30 02:00:03.995442828 +0000 UTC m=+1882.498375047" Oct 30 02:00:04.039847 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.039791 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" podStartSLOduration=4.039775768 podStartE2EDuration="4.039775768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:04.038995275 +0000 UTC m=+1882.541927496" watchObservedRunningTime="2026-10-30 02:00:04.039775768 +0000 UTC m=+1882.542707987" Oct 30 02:00:04.884978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.884935 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" event={"ID":"b822b327-db3b-459c-943a-39691f3875a1","Type":"ContainerStarted","Data":"c049e4aaca304d0b79a6e4785211a09dabdc5fc8fd36dee60c51abf59c0aa654"} Oct 30 02:00:04.887069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.887042 5035 generic.go:356] "Generic (PLEG): container finished" podID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerID="cc2b8c9d2f4d3bcfa0483e6a9e340eac9133201e9e49fc1bcedeb4ac48405bdc" exitCode=60 Oct 30 02:00:04.887217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.887126 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" event={"ID":"63e725b7-a078-45ca-aac5-1b0f5fcc177d","Type":"ContainerDied","Data":"cc2b8c9d2f4d3bcfa0483e6a9e340eac9133201e9e49fc1bcedeb4ac48405bdc"} Oct 30 02:00:04.887217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.887188 5035 scope.go:122] "RemoveContainer" containerID="bcecdd6208bf5145b929b0f7fe832796c26589ff3014f62814b030bdc97a6889" Oct 30 02:00:04.887444 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.887423 5035 scope.go:122] "RemoveContainer" containerID="cc2b8c9d2f4d3bcfa0483e6a9e340eac9133201e9e49fc1bcedeb4ac48405bdc" Oct 30 02:00:04.887870 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:04.887847 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz_odh-ai-gateway-infra(63e725b7-a078-45ca-aac5-1b0f5fcc177d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" Oct 30 02:00:04.889288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.889260 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" event={"ID":"14084516-d9fb-4b98-8609-3f55b46688e4","Type":"ContainerStarted","Data":"68ef9ba928390091d90487b9725187ae3eb4aa612057a1c6a8a00a719b351f49"} Oct 30 02:00:04.891271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.891242 5035 generic.go:356] "Generic (PLEG): container finished" podID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerID="fc16297508a87b39cd9b53913b5d6d8cea8daf44e40b78ac236dec55c737f477" exitCode=60 Oct 30 02:00:04.891374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.891282 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" event={"ID":"6d14078e-51dc-4bcf-9a9c-17ee7901be40","Type":"ContainerDied","Data":"fc16297508a87b39cd9b53913b5d6d8cea8daf44e40b78ac236dec55c737f477"} Oct 30 02:00:04.891639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.891620 5035 scope.go:122] "RemoveContainer" containerID="fc16297508a87b39cd9b53913b5d6d8cea8daf44e40b78ac236dec55c737f477" Oct 30 02:00:04.892048 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:04.892024 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh_odh-ai-gateway-infra(6d14078e-51dc-4bcf-9a9c-17ee7901be40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" Oct 30 02:00:04.893355 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.893333 5035 generic.go:356] "Generic (PLEG): container finished" podID="0214eaff-b6f8-491f-9259-18431f48a390" containerID="7fa054c9988d1c688488e2844e1bdee1a4741f3f21896509539a184961801810" exitCode=60 Oct 30 02:00:04.893460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.893399 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" event={"ID":"0214eaff-b6f8-491f-9259-18431f48a390","Type":"ContainerDied","Data":"7fa054c9988d1c688488e2844e1bdee1a4741f3f21896509539a184961801810"} Oct 30 02:00:04.893766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.893748 5035 scope.go:122] "RemoveContainer" containerID="7fa054c9988d1c688488e2844e1bdee1a4741f3f21896509539a184961801810" Oct 30 02:00:04.894123 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:04.894100 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm_odh-ai-gateway-infra(0214eaff-b6f8-491f-9259-18431f48a390)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" podUID="0214eaff-b6f8-491f-9259-18431f48a390" Oct 30 02:00:04.901939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.901915 5035 scope.go:122] "RemoveContainer" containerID="4a167a589f69b3ccfb888d1f69be43b9a2d08a55e8da6495265da36dde8922ce" Oct 30 02:00:04.917295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:04.917269 5035 scope.go:122] "RemoveContainer" containerID="208bd414a900d709ae32c787908983a56ead8a8a0a23adf489a4246a5650a361" Oct 30 02:00:05.903817 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.903779 5035 scope.go:122] "RemoveContainer" containerID="7fa054c9988d1c688488e2844e1bdee1a4741f3f21896509539a184961801810" Oct 30 02:00:05.904340 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:05.904218 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm_odh-ai-gateway-infra(0214eaff-b6f8-491f-9259-18431f48a390)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" podUID="0214eaff-b6f8-491f-9259-18431f48a390" Oct 30 02:00:05.905757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.905728 5035 generic.go:356] "Generic (PLEG): container finished" podID="b822b327-db3b-459c-943a-39691f3875a1" containerID="c049e4aaca304d0b79a6e4785211a09dabdc5fc8fd36dee60c51abf59c0aa654" exitCode=60 Oct 30 02:00:05.905884 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.905800 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" event={"ID":"b822b327-db3b-459c-943a-39691f3875a1","Type":"ContainerDied","Data":"c049e4aaca304d0b79a6e4785211a09dabdc5fc8fd36dee60c51abf59c0aa654"} Oct 30 02:00:05.905884 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.905845 5035 scope.go:122] "RemoveContainer" containerID="63b8bdbb1046267fdc392618cb49714ca5aaa9a5e4e53d7b7c9c7fd43c1bfbb6" Oct 30 02:00:05.906225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.906208 5035 scope.go:122] "RemoveContainer" containerID="c049e4aaca304d0b79a6e4785211a09dabdc5fc8fd36dee60c51abf59c0aa654" Oct 30 02:00:05.906586 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:05.906562 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s_odh-ai-gateway-infra(b822b327-db3b-459c-943a-39691f3875a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" podUID="b822b327-db3b-459c-943a-39691f3875a1" Oct 30 02:00:05.909255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.909227 5035 scope.go:122] "RemoveContainer" containerID="cc2b8c9d2f4d3bcfa0483e6a9e340eac9133201e9e49fc1bcedeb4ac48405bdc" Oct 30 02:00:05.909643 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:05.909610 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz_odh-ai-gateway-infra(63e725b7-a078-45ca-aac5-1b0f5fcc177d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" Oct 30 02:00:05.910954 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.910931 5035 generic.go:356] "Generic (PLEG): container finished" podID="14084516-d9fb-4b98-8609-3f55b46688e4" containerID="68ef9ba928390091d90487b9725187ae3eb4aa612057a1c6a8a00a719b351f49" exitCode=60 Oct 30 02:00:05.911034 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.911011 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" event={"ID":"14084516-d9fb-4b98-8609-3f55b46688e4","Type":"ContainerDied","Data":"68ef9ba928390091d90487b9725187ae3eb4aa612057a1c6a8a00a719b351f49"} Oct 30 02:00:05.911273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.911259 5035 scope.go:122] "RemoveContainer" containerID="68ef9ba928390091d90487b9725187ae3eb4aa612057a1c6a8a00a719b351f49" Oct 30 02:00:05.911657 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:05.911631 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs_odh-ai-gateway-infra(14084516-d9fb-4b98-8609-3f55b46688e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" Oct 30 02:00:05.913017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.912997 5035 scope.go:122] "RemoveContainer" containerID="fc16297508a87b39cd9b53913b5d6d8cea8daf44e40b78ac236dec55c737f477" Oct 30 02:00:05.913264 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:05.913247 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh_odh-ai-gateway-infra(6d14078e-51dc-4bcf-9a9c-17ee7901be40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" Oct 30 02:00:05.921599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:05.921573 5035 scope.go:122] "RemoveContainer" containerID="c624369ae042c591e2bf5d77680eea4a919cbc8fd7cb4fda37649b081f07925d" Oct 30 02:00:06.924289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:06.924253 5035 scope.go:122] "RemoveContainer" containerID="c049e4aaca304d0b79a6e4785211a09dabdc5fc8fd36dee60c51abf59c0aa654" Oct 30 02:00:06.924834 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:06.924729 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s_odh-ai-gateway-infra(b822b327-db3b-459c-943a-39691f3875a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" podUID="b822b327-db3b-459c-943a-39691f3875a1" Oct 30 02:00:06.926211 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:06.926189 5035 scope.go:122] "RemoveContainer" containerID="68ef9ba928390091d90487b9725187ae3eb4aa612057a1c6a8a00a719b351f49" Oct 30 02:00:06.926539 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:06.926507 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs_odh-ai-gateway-infra(14084516-d9fb-4b98-8609-3f55b46688e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" Oct 30 02:00:07.932876 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.932824 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx"] Oct 30 02:00:07.933567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933550 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" Oct 30 02:00:07.933623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933568 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" Oct 30 02:00:07.933623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933583 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" Oct 30 02:00:07.933623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933589 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" Oct 30 02:00:07.933623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933599 5035 cpu_manager.go:414] "containerMap: removing container" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" Oct 30 02:00:07.933623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933606 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" Oct 30 02:00:07.933785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933662 5035 cpu_manager.go:414] "containerMap: removing container" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" Oct 30 02:00:07.933785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933667 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" Oct 30 02:00:07.933785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933764 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5083785-8e42-4372-8d43-49cc6edceda4" containerName="istio-proxy" Oct 30 02:00:07.933785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933783 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9e1898-ff05-4c0c-bf18-858d74c2b636" containerName="istio-proxy" Oct 30 02:00:07.933905 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933797 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ce2c95-167f-40f3-9747-f411d0bc8ea5" containerName="istio-proxy" Oct 30 02:00:07.933905 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.933807 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07e40cb-209d-47ee-9324-cd445c485cb6" containerName="istio-proxy" Oct 30 02:00:07.938809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.938786 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:07.963506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.963467 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx"] Oct 30 02:00:07.969823 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.969782 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-356082\"" Oct 30 02:00:07.969823 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:07.969785 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-356082-dockercfg-z4lwn\"" Oct 30 02:00:08.038554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.026219 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh"] Oct 30 02:00:08.124967 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.124891 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx\" (UID: \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:08.125590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.125471 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5sc\" (UniqueName: \"kubernetes.io/projected/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-kube-api-access-8z5sc\") pod \"payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx\" (UID: \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:08.183518 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.171070 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:08.183518 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.180027 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh"] Oct 30 02:00:08.227445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.227389 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5sc\" (UniqueName: \"kubernetes.io/projected/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-kube-api-access-8z5sc\") pod \"payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx\" (UID: \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:08.227445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.227447 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9cnv\" (UniqueName: \"kubernetes.io/projected/c0540264-0f4b-4a01-82db-5eb14a142273-kube-api-access-g9cnv\") pod \"payload-processing-e2e-api-b-356082-84fd4794fb-578fh\" (UID: \"c0540264-0f4b-4a01-82db-5eb14a142273\") " pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:08.227706 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.227472 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0540264-0f4b-4a01-82db-5eb14a142273-plugins-config-volume\") pod \"payload-processing-e2e-api-b-356082-84fd4794fb-578fh\" (UID: \"c0540264-0f4b-4a01-82db-5eb14a142273\") " pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:08.227826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.227707 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx\" (UID: \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:08.228511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.228489 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx\" (UID: \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:08.252082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.252049 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5sc\" (UniqueName: \"kubernetes.io/projected/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-kube-api-access-8z5sc\") pod \"payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx\" (UID: \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:08.254928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.254895 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:08.329907 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.329864 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9cnv\" (UniqueName: \"kubernetes.io/projected/c0540264-0f4b-4a01-82db-5eb14a142273-kube-api-access-g9cnv\") pod \"payload-processing-e2e-api-b-356082-84fd4794fb-578fh\" (UID: \"c0540264-0f4b-4a01-82db-5eb14a142273\") " pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:08.330042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.329929 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0540264-0f4b-4a01-82db-5eb14a142273-plugins-config-volume\") pod \"payload-processing-e2e-api-b-356082-84fd4794fb-578fh\" (UID: \"c0540264-0f4b-4a01-82db-5eb14a142273\") " pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:08.331022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.330996 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0540264-0f4b-4a01-82db-5eb14a142273-plugins-config-volume\") pod \"payload-processing-e2e-api-b-356082-84fd4794fb-578fh\" (UID: \"c0540264-0f4b-4a01-82db-5eb14a142273\") " pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:08.363495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.363462 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9cnv\" (UniqueName: \"kubernetes.io/projected/c0540264-0f4b-4a01-82db-5eb14a142273-kube-api-access-g9cnv\") pod \"payload-processing-e2e-api-b-356082-84fd4794fb-578fh\" (UID: \"c0540264-0f4b-4a01-82db-5eb14a142273\") " pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:08.495316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.495275 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:08.579562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.577752 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx"] Oct 30 02:00:08.583811 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:08.583769 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c5206e_3ba7_4478_98a5_5dae3ff8f11d.slice/crio-4ebb26c598c66fd9fb5b1ad0ac3205ce26c5694719f62007e86614a681852e52 WatchSource:0}: Error finding container 4ebb26c598c66fd9fb5b1ad0ac3205ce26c5694719f62007e86614a681852e52: Status 404 returned error can't find the container with id 4ebb26c598c66fd9fb5b1ad0ac3205ce26c5694719f62007e86614a681852e52 Oct 30 02:00:08.812387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.812355 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh"] Oct 30 02:00:08.813676 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:08.813645 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0540264_0f4b_4a01_82db_5eb14a142273.slice/crio-f8a6301ff92f03201f0f7669594cbecee2cee957b004fa9bd62e8ec509e429d6 WatchSource:0}: Error finding container f8a6301ff92f03201f0f7669594cbecee2cee957b004fa9bd62e8ec509e429d6: Status 404 returned error can't find the container with id f8a6301ff92f03201f0f7669594cbecee2cee957b004fa9bd62e8ec509e429d6 Oct 30 02:00:08.945767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.945725 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" event={"ID":"c0540264-0f4b-4a01-82db-5eb14a142273","Type":"ContainerStarted","Data":"bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff"} Oct 30 02:00:08.946259 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.945797 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" event={"ID":"c0540264-0f4b-4a01-82db-5eb14a142273","Type":"ContainerStarted","Data":"f8a6301ff92f03201f0f7669594cbecee2cee957b004fa9bd62e8ec509e429d6"} Oct 30 02:00:08.946259 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.945830 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:08.947711 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.947678 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" event={"ID":"90c5206e-3ba7-4478-98a5-5dae3ff8f11d","Type":"ContainerStarted","Data":"db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8"} Oct 30 02:00:08.947865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.947741 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:08.947865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.947766 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" event={"ID":"90c5206e-3ba7-4478-98a5-5dae3ff8f11d","Type":"ContainerStarted","Data":"4ebb26c598c66fd9fb5b1ad0ac3205ce26c5694719f62007e86614a681852e52"} Oct 30 02:00:08.989697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:08.989640 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" podStartSLOduration=1.989621633 podStartE2EDuration="1.989621633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:08.988047037 +0000 UTC m=+1887.490979257" watchObservedRunningTime="2026-10-30 02:00:08.989621633 +0000 UTC m=+1887.492553878" Oct 30 02:00:09.034505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:09.034448 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" podStartSLOduration=2.034433388 podStartE2EDuration="2.034433388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:09.03075077 +0000 UTC m=+1887.533682991" watchObservedRunningTime="2026-10-30 02:00:09.034433388 +0000 UTC m=+1887.537365607" Oct 30 02:00:14.635724 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:14.635691 5035 scope.go:122] "RemoveContainer" containerID="cc2b8c9d2f4d3bcfa0483e6a9e340eac9133201e9e49fc1bcedeb4ac48405bdc" Oct 30 02:00:15.730433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.730377 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr"] Oct 30 02:00:15.735490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.735459 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.748978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.748940 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn"] Oct 30 02:00:15.753938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.753904 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.759778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.759745 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6ba1d9-openshift-default-dockercfg-67skz\"" Oct 30 02:00:15.760932 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.760900 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr"] Oct 30 02:00:15.787015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.786980 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn"] Oct 30 02:00:15.842793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.842757 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27942cd1-bb9c-4470-89d2-89d82022090a-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.842979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.842803 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.842979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.842885 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.842979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.842921 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.843090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843061 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.843123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843102 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.843188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843175 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzn7\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-kube-api-access-6hzn7\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.843221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843205 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.843258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843233 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.843289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843272 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.843370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843357 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.843405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843387 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27942cd1-bb9c-4470-89d2-89d82022090a-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.843442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843408 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.843520 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843508 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfv2\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-kube-api-access-tbfv2\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.843568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843547 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.843606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843575 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.843641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843622 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.843679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.843656 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.945399 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945355 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfv2\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-kube-api-access-tbfv2\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.945629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945410 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.945629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945440 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.945629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945469 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.945629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945567 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.945629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945620 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27942cd1-bb9c-4470-89d2-89d82022090a-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.945911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945651 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.945911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945684 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.945911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945713 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.945911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945836 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.945911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945883 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.946160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945934 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzn7\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-kube-api-access-6hzn7\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.946160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945974 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.946160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945980 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.946160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946021 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.946160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946038 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.946160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.945980 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.946160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946087 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.946502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946199 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.946502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946233 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.946502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946250 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27942cd1-bb9c-4470-89d2-89d82022090a-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.946502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946280 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.946502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946343 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.946502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946488 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.946908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946560 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.946908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946574 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27942cd1-bb9c-4470-89d2-89d82022090a-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.946908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946783 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.946908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.946858 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.948676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.948644 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.948811 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.948733 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.949023 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.949001 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.949111 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.949052 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27942cd1-bb9c-4470-89d2-89d82022090a-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.971230 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.971178 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfv2\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-kube-api-access-tbfv2\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.971426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.971302 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:15.971426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.971346 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:15.971558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:15.971477 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzn7\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-kube-api-access-6hzn7\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:16.015481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.015382 5035 generic.go:356] "Generic (PLEG): container finished" podID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerID="c23ad47c60aba7488fa0667017d1640c6550366efea5a6c70c64600d4681065a" exitCode=60 Oct 30 02:00:16.015481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.015446 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" event={"ID":"63e725b7-a078-45ca-aac5-1b0f5fcc177d","Type":"ContainerDied","Data":"c23ad47c60aba7488fa0667017d1640c6550366efea5a6c70c64600d4681065a"} Oct 30 02:00:16.015731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.015575 5035 scope.go:122] "RemoveContainer" containerID="cc2b8c9d2f4d3bcfa0483e6a9e340eac9133201e9e49fc1bcedeb4ac48405bdc" Oct 30 02:00:16.016259 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.016233 5035 scope.go:122] "RemoveContainer" containerID="c23ad47c60aba7488fa0667017d1640c6550366efea5a6c70c64600d4681065a" Oct 30 02:00:16.016737 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:16.016710 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz_odh-ai-gateway-infra(63e725b7-a078-45ca-aac5-1b0f5fcc177d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" Oct 30 02:00:16.053003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.052972 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:16.089343 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.089293 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:16.394613 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.394579 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr"] Oct 30 02:00:16.395047 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:16.395021 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27942cd1_bb9c_4470_89d2_89d82022090a.slice/crio-8f656e0d5e389dba0323fadb9b3a6427335f7278e90acc3498883341f8c64b83 WatchSource:0}: Error finding container 8f656e0d5e389dba0323fadb9b3a6427335f7278e90acc3498883341f8c64b83: Status 404 returned error can't find the container with id 8f656e0d5e389dba0323fadb9b3a6427335f7278e90acc3498883341f8c64b83 Oct 30 02:00:16.398035 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.398002 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:16.398121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.398068 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:16.398121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.398101 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:16.444002 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:16.443972 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2c413f_3ed0_4f81_9c68_ba40983d4ba6.slice/crio-00ab2fc7a0ba3d9c7a7333ef4c1c7b9ae0231e333500adc449d32184d6649fa7 WatchSource:0}: Error finding container 00ab2fc7a0ba3d9c7a7333ef4c1c7b9ae0231e333500adc449d32184d6649fa7: Status 404 returned error can't find the container with id 00ab2fc7a0ba3d9c7a7333ef4c1c7b9ae0231e333500adc449d32184d6649fa7 Oct 30 02:00:16.447833 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.447798 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn"] Oct 30 02:00:16.448287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.448240 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:16.448382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.448341 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:16.448450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.448387 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:16.541726 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.541682 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss"] Oct 30 02:00:16.571440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.571224 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l"] Oct 30 02:00:16.639551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:16.636358 5035 scope.go:122] "RemoveContainer" containerID="fc16297508a87b39cd9b53913b5d6d8cea8daf44e40b78ac236dec55c737f477" Oct 30 02:00:17.026072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.026024 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" event={"ID":"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6","Type":"ContainerStarted","Data":"f91cf31fcd06e8fe92e66d3f7f4efc1173394ecf5666dcc3ab19942f290b34dc"} Oct 30 02:00:17.026741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.026099 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" event={"ID":"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6","Type":"ContainerStarted","Data":"00ab2fc7a0ba3d9c7a7333ef4c1c7b9ae0231e333500adc449d32184d6649fa7"} Oct 30 02:00:17.030101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.030065 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" event={"ID":"27942cd1-bb9c-4470-89d2-89d82022090a","Type":"ContainerStarted","Data":"06ba1b180bb62e556df1b79f5671fa0c0fb6da2efe35290482101925c19954be"} Oct 30 02:00:17.030259 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.030135 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" event={"ID":"27942cd1-bb9c-4470-89d2-89d82022090a","Type":"ContainerStarted","Data":"8f656e0d5e389dba0323fadb9b3a6427335f7278e90acc3498883341f8c64b83"} Oct 30 02:00:17.053755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.053718 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:17.055137 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.055113 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 02:00:17.055205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.055162 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 02:00:17.080454 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.080393 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" podStartSLOduration=2.080372527 podStartE2EDuration="2.080372527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:17.076938497 +0000 UTC m=+1895.579870718" watchObservedRunningTime="2026-10-30 02:00:17.080372527 +0000 UTC m=+1895.583304750" Oct 30 02:00:17.089467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.089436 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:17.091393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.091338 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:00:17.091633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.091401 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 02:00:17.095603 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.095566 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz"] Oct 30 02:00:17.135490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.132749 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" podStartSLOduration=2.132731085 podStartE2EDuration="2.132731085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:17.131673615 +0000 UTC m=+1895.634605865" watchObservedRunningTime="2026-10-30 02:00:17.132731085 +0000 UTC m=+1895.635663307" Oct 30 02:00:17.620165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.620124 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k"] Oct 30 02:00:17.634266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.634235 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:17.636729 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.636698 5035 scope.go:122] "RemoveContainer" containerID="7fa054c9988d1c688488e2844e1bdee1a4741f3f21896509539a184961801810" Oct 30 02:00:17.637486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.637444 5035 scope.go:122] "RemoveContainer" containerID="68ef9ba928390091d90487b9725187ae3eb4aa612057a1c6a8a00a719b351f49" Oct 30 02:00:17.638296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.638280 5035 scope.go:122] "RemoveContainer" containerID="c049e4aaca304d0b79a6e4785211a09dabdc5fc8fd36dee60c51abf59c0aa654" Oct 30 02:00:17.659408 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:17.659363 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:00:17.755309 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.755172 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e725b7-a078-45ca-aac5-1b0f5fcc177d-kube-api-access-5jc2z\" (UniqueName: \"kubernetes.io/projected/63e725b7-a078-45ca-aac5-1b0f5fcc177d-kube-api-access-5jc2z\") pod \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\" (UID: \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\") " Oct 30 02:00:17.755503 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.755466 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e725b7-a078-45ca-aac5-1b0f5fcc177d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e725b7-a078-45ca-aac5-1b0f5fcc177d-maas-api-service-ca\") pod \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\" (UID: \"63e725b7-a078-45ca-aac5-1b0f5fcc177d\") " Oct 30 02:00:17.756792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.756034 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e725b7-a078-45ca-aac5-1b0f5fcc177d-maas-api-service-ca" pod "63e725b7-a078-45ca-aac5-1b0f5fcc177d" (UID: "63e725b7-a078-45ca-aac5-1b0f5fcc177d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:17.756792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.756333 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e725b7-a078-45ca-aac5-1b0f5fcc177d-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:17.766194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.766130 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e725b7-a078-45ca-aac5-1b0f5fcc177d-kube-api-access-5jc2z" pod "63e725b7-a078-45ca-aac5-1b0f5fcc177d" (UID: "63e725b7-a078-45ca-aac5-1b0f5fcc177d"). InnerVolumeSpecName "kube-api-access-5jc2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:17.965302 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:17.897619 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jc2z\" (UniqueName: \"kubernetes.io/projected/63e725b7-a078-45ca-aac5-1b0f5fcc177d-kube-api-access-5jc2z\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:18.054363 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.054056 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 02:00:18.054363 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.054122 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 02:00:18.057709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.057673 5035 generic.go:356] "Generic (PLEG): container finished" podID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerID="77c870556983182170a984362576602e396b1a613f235509c1444fe147d597ba" exitCode=60 Oct 30 02:00:18.057926 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.057743 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" event={"ID":"6d14078e-51dc-4bcf-9a9c-17ee7901be40","Type":"ContainerDied","Data":"77c870556983182170a984362576602e396b1a613f235509c1444fe147d597ba"} Oct 30 02:00:18.057926 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.057807 5035 scope.go:122] "RemoveContainer" containerID="fc16297508a87b39cd9b53913b5d6d8cea8daf44e40b78ac236dec55c737f477" Oct 30 02:00:18.058265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.058241 5035 scope.go:122] "RemoveContainer" containerID="77c870556983182170a984362576602e396b1a613f235509c1444fe147d597ba" Oct 30 02:00:18.058677 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:18.058653 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh_odh-ai-gateway-infra(6d14078e-51dc-4bcf-9a9c-17ee7901be40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" Oct 30 02:00:18.060568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.060429 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" Oct 30 02:00:18.060568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.060424 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz" event={"ID":"63e725b7-a078-45ca-aac5-1b0f5fcc177d","Type":"ContainerDied","Data":"b7098f200ead09cd9a9c12a4861df07cf410ec3cdb70071f252ae1dac3072a4c"} Oct 30 02:00:18.093555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.090397 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:00:18.094365 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.094302 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 02:00:18.160050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.158865 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx"] Oct 30 02:00:18.160050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.158927 5035 scope.go:122] "RemoveContainer" containerID="c23ad47c60aba7488fa0667017d1640c6550366efea5a6c70c64600d4681065a" Oct 30 02:00:18.160050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.159930 5035 cpu_manager.go:414] "containerMap: removing container" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerName="cleanup" Oct 30 02:00:18.160050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.159951 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerName="cleanup" Oct 30 02:00:18.160050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.159992 5035 cpu_manager.go:414] "containerMap: removing container" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerName="cleanup" Oct 30 02:00:18.160050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.160001 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerName="cleanup" Oct 30 02:00:18.160050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.160024 5035 cpu_manager.go:414] "containerMap: removing container" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerName="cleanup" Oct 30 02:00:18.160050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.160033 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerName="cleanup" Oct 30 02:00:18.187925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.160254 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerName="cleanup" Oct 30 02:00:18.187925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.160272 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerName="cleanup" Oct 30 02:00:18.187925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.160285 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" containerName="cleanup" Oct 30 02:00:18.187925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.166062 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.223635 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.196961 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-145336-openshift-default-dockercfg-n2t9b\"" Oct 30 02:00:18.247753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.247717 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz"] Oct 30 02:00:18.254630 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.254587 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx"] Oct 30 02:00:18.263767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.263735 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bd098e-29888760-74mkz"] Oct 30 02:00:18.360478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.360424 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-credential-socket\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.360778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.360750 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0da372-b63d-4050-899c-eb5007647157-istio-podinfo\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.360891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.360790 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-certs\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.360891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.360817 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0da372-b63d-4050-899c-eb5007647157-istiod-ca-cert\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.361020 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.360993 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-istio-token\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.361123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.361104 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-socket\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.361172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.361161 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-envoy\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.361265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.361253 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-data\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.361316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.361278 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tqd\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-kube-api-access-v2tqd\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.480117 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-certs\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.480192 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0da372-b63d-4050-899c-eb5007647157-istiod-ca-cert\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.480283 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-istio-token\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.480393 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-socket\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.480466 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-envoy\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.480601 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-data\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.480651 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tqd\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-kube-api-access-v2tqd\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.480851 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-credential-socket\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.480933 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0da372-b63d-4050-899c-eb5007647157-istio-podinfo\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.481999 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-socket\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.482370 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-certs\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.484773 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-data\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.486993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.485038 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-credential-socket\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.518549 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.516508 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0da372-b63d-4050-899c-eb5007647157-istiod-ca-cert\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.555558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.551423 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0da372-b63d-4050-899c-eb5007647157-istio-podinfo\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.567576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.558121 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-istio-token\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.590638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.572266 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tqd\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-kube-api-access-v2tqd\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.602578 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.590878 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-envoy\") pod \"e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:18.682037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:18.681992 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:19.053692 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.053648 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 02:00:19.053885 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.053714 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 02:00:19.127053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.124167 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:00:19.127053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.124242 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 02:00:19.128926 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.128496 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" event={"ID":"14084516-d9fb-4b98-8609-3f55b46688e4","Type":"ContainerStarted","Data":"66926b3f9652cccf599eb7a325eb8ba5ab62e84aa46c1e9b38ed591f2ed7176d"} Oct 30 02:00:19.128926 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.128595 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh"] Oct 30 02:00:19.131405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.131367 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" event={"ID":"b822b327-db3b-459c-943a-39691f3875a1","Type":"ContainerStarted","Data":"8eb6ff8fea539c66eb573d48727c43509ae6d2d6aab1856f301c379656ce1093"} Oct 30 02:00:19.133482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.133450 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx"] Oct 30 02:00:19.137276 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.136939 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" event={"ID":"0214eaff-b6f8-491f-9259-18431f48a390","Type":"ContainerStarted","Data":"c7cb60082b9204f9df77d57fb1efe4aeda9636ffe887c1ee7455f55c8f02593d"} Oct 30 02:00:19.141558 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:19.141476 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0da372_b63d_4050_899c_eb5007647157.slice/crio-ba654e83010589dfaf0f810be3ca17d636f8d55d7114914b2c3c84c291693526 WatchSource:0}: Error finding container ba654e83010589dfaf0f810be3ca17d636f8d55d7114914b2c3c84c291693526: Status 404 returned error can't find the container with id ba654e83010589dfaf0f810be3ca17d636f8d55d7114914b2c3c84c291693526 Oct 30 02:00:19.144244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.144195 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:19.144354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.144284 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:19.144354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.144329 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:19.448733 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.448707 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:19.559715 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.559319 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d14078e-51dc-4bcf-9a9c-17ee7901be40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d14078e-51dc-4bcf-9a9c-17ee7901be40-maas-api-service-ca\") pod \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\" (UID: \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\") " Oct 30 02:00:19.559950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.559849 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d14078e-51dc-4bcf-9a9c-17ee7901be40-maas-api-service-ca" pod "6d14078e-51dc-4bcf-9a9c-17ee7901be40" (UID: "6d14078e-51dc-4bcf-9a9c-17ee7901be40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:19.560045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.559972 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d14078e-51dc-4bcf-9a9c-17ee7901be40-kube-api-access-p98zl\" (UniqueName: \"kubernetes.io/projected/6d14078e-51dc-4bcf-9a9c-17ee7901be40-kube-api-access-p98zl\") pod \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\" (UID: \"6d14078e-51dc-4bcf-9a9c-17ee7901be40\") " Oct 30 02:00:19.560656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.560628 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d14078e-51dc-4bcf-9a9c-17ee7901be40-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:19.564075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.564042 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d14078e-51dc-4bcf-9a9c-17ee7901be40-kube-api-access-p98zl" pod "6d14078e-51dc-4bcf-9a9c-17ee7901be40" (UID: "6d14078e-51dc-4bcf-9a9c-17ee7901be40"). InnerVolumeSpecName "kube-api-access-p98zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:19.655236 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.655196 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e725b7-a078-45ca-aac5-1b0f5fcc177d" path="/var/lib/kubelet/pods/63e725b7-a078-45ca-aac5-1b0f5fcc177d/volumes" Oct 30 02:00:19.662226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.662189 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p98zl\" (UniqueName: \"kubernetes.io/projected/6d14078e-51dc-4bcf-9a9c-17ee7901be40-kube-api-access-p98zl\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:19.958972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.958858 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:00:19.959506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:19.959474 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:00:20.054346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.054247 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 02:00:20.054346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.054317 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 02:00:20.120558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.119630 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:00:20.120558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.119693 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 02:00:20.158471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.158220 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" Oct 30 02:00:20.158471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.158225 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh" event={"ID":"6d14078e-51dc-4bcf-9a9c-17ee7901be40","Type":"ContainerDied","Data":"e01a4a008989fe77f1519b1a66cab4cf047e50bc00fb0fef650ddcffa3d77aa4"} Oct 30 02:00:20.158471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.158351 5035 scope.go:122] "RemoveContainer" containerID="77c870556983182170a984362576602e396b1a613f235509c1444fe147d597ba" Oct 30 02:00:20.163003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.162972 5035 generic.go:356] "Generic (PLEG): container finished" podID="0214eaff-b6f8-491f-9259-18431f48a390" containerID="c7cb60082b9204f9df77d57fb1efe4aeda9636ffe887c1ee7455f55c8f02593d" exitCode=60 Oct 30 02:00:20.163320 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.163145 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" event={"ID":"0214eaff-b6f8-491f-9259-18431f48a390","Type":"ContainerDied","Data":"c7cb60082b9204f9df77d57fb1efe4aeda9636ffe887c1ee7455f55c8f02593d"} Oct 30 02:00:20.163990 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.163968 5035 scope.go:122] "RemoveContainer" containerID="c7cb60082b9204f9df77d57fb1efe4aeda9636ffe887c1ee7455f55c8f02593d" Oct 30 02:00:20.164560 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:20.164502 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm_odh-ai-gateway-infra(0214eaff-b6f8-491f-9259-18431f48a390)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" podUID="0214eaff-b6f8-491f-9259-18431f48a390" Oct 30 02:00:20.166600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.166509 5035 generic.go:356] "Generic (PLEG): container finished" podID="14084516-d9fb-4b98-8609-3f55b46688e4" containerID="66926b3f9652cccf599eb7a325eb8ba5ab62e84aa46c1e9b38ed591f2ed7176d" exitCode=60 Oct 30 02:00:20.179849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.174841 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" event={"ID":"14084516-d9fb-4b98-8609-3f55b46688e4","Type":"ContainerDied","Data":"66926b3f9652cccf599eb7a325eb8ba5ab62e84aa46c1e9b38ed591f2ed7176d"} Oct 30 02:00:20.243245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.243184 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" event={"ID":"0a0da372-b63d-4050-899c-eb5007647157","Type":"ContainerStarted","Data":"145e6d1c1b67436098570cd37cee3ada549268458362fd9c43fe0ad96095a0e5"} Oct 30 02:00:20.243245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.243239 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" event={"ID":"0a0da372-b63d-4050-899c-eb5007647157","Type":"ContainerStarted","Data":"ba654e83010589dfaf0f810be3ca17d636f8d55d7114914b2c3c84c291693526"} Oct 30 02:00:20.243580 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.243266 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs"] Oct 30 02:00:20.245265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.245234 5035 scope.go:122] "RemoveContainer" containerID="7fa054c9988d1c688488e2844e1bdee1a4741f3f21896509539a184961801810" Oct 30 02:00:20.246799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.246767 5035 generic.go:356] "Generic (PLEG): container finished" podID="b822b327-db3b-459c-943a-39691f3875a1" containerID="8eb6ff8fea539c66eb573d48727c43509ae6d2d6aab1856f301c379656ce1093" exitCode=60 Oct 30 02:00:20.247079 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.247057 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh"] Oct 30 02:00:20.247212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.247106 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" event={"ID":"b822b327-db3b-459c-943a-39691f3875a1","Type":"ContainerDied","Data":"8eb6ff8fea539c66eb573d48727c43509ae6d2d6aab1856f301c379656ce1093"} Oct 30 02:00:20.247611 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.247584 5035 scope.go:122] "RemoveContainer" containerID="8eb6ff8fea539c66eb573d48727c43509ae6d2d6aab1856f301c379656ce1093" Oct 30 02:00:20.248023 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:20.247997 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s_odh-ai-gateway-infra(b822b327-db3b-459c-943a-39691f3875a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" podUID="b822b327-db3b-459c-943a-39691f3875a1" Oct 30 02:00:20.253811 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.253773 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b273b6-29888760-8jfdh"] Oct 30 02:00:20.266296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.266256 5035 scope.go:122] "RemoveContainer" containerID="68ef9ba928390091d90487b9725187ae3eb4aa612057a1c6a8a00a719b351f49" Oct 30 02:00:20.347637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.347604 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s"] Oct 30 02:00:20.387393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.387256 5035 scope.go:122] "RemoveContainer" containerID="c049e4aaca304d0b79a6e4785211a09dabdc5fc8fd36dee60c51abf59c0aa654" Oct 30 02:00:20.442893 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.441576 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm"] Oct 30 02:00:20.501613 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.494563 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" podStartSLOduration=2.494519952 podStartE2EDuration="2.494519952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:20.492713556 +0000 UTC m=+1898.995645814" watchObservedRunningTime="2026-10-30 02:00:20.494519952 +0000 UTC m=+1898.997452173" Oct 30 02:00:20.742814 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.742757 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:20.748452 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.748425 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:20.825390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.825353 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:20.857131 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.857088 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14084516-d9fb-4b98-8609-3f55b46688e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14084516-d9fb-4b98-8609-3f55b46688e4-maas-api-service-ca\") pod \"14084516-d9fb-4b98-8609-3f55b46688e4\" (UID: \"14084516-d9fb-4b98-8609-3f55b46688e4\") " Oct 30 02:00:20.857353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.857331 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14084516-d9fb-4b98-8609-3f55b46688e4-kube-api-access-nkg5h\" (UniqueName: \"kubernetes.io/projected/14084516-d9fb-4b98-8609-3f55b46688e4-kube-api-access-nkg5h\") pod \"14084516-d9fb-4b98-8609-3f55b46688e4\" (UID: \"14084516-d9fb-4b98-8609-3f55b46688e4\") " Oct 30 02:00:20.857517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.857489 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14084516-d9fb-4b98-8609-3f55b46688e4-maas-api-service-ca" pod "14084516-d9fb-4b98-8609-3f55b46688e4" (UID: "14084516-d9fb-4b98-8609-3f55b46688e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:20.858056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.858039 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14084516-d9fb-4b98-8609-3f55b46688e4-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:20.860588 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.860547 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14084516-d9fb-4b98-8609-3f55b46688e4-kube-api-access-nkg5h" pod "14084516-d9fb-4b98-8609-3f55b46688e4" (UID: "14084516-d9fb-4b98-8609-3f55b46688e4"). InnerVolumeSpecName "kube-api-access-nkg5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:20.959590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:20.959543 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkg5h\" (UniqueName: \"kubernetes.io/projected/14084516-d9fb-4b98-8609-3f55b46688e4-kube-api-access-nkg5h\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.053617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.053469 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 02:00:21.053617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.053544 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 02:00:21.090557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.090049 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:00:21.090557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.090121 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 02:00:21.330293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.330265 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" Oct 30 02:00:21.330717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.330359 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs" event={"ID":"14084516-d9fb-4b98-8609-3f55b46688e4","Type":"ContainerDied","Data":"a8d6b501eb23b9f7c3395e058b073fccebc38d8a495003f83735bdb35c99206b"} Oct 30 02:00:21.330717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.330425 5035 scope.go:122] "RemoveContainer" containerID="66926b3f9652cccf599eb7a325eb8ba5ab62e84aa46c1e9b38ed591f2ed7176d" Oct 30 02:00:21.331591 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.331569 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:21.332864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.332831 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:00:21.433537 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.433475 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs"] Oct 30 02:00:21.441954 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.441914 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-86dd79-29888760-8clqs"] Oct 30 02:00:21.542631 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.542385 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" podUID="6b9160fd-4164-484f-b5bb-88f8846934d4" containerName="payload-processing" containerID="cri-o://ec3bce30a5a5cfa4f2bd7d20ff4b9a45daab62817deddeee648928d9d1c74dae" gracePeriod=25 Oct 30 02:00:21.598629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.581609 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" podUID="04b2d88d-c92d-49e6-84d6-e7267ae2b7fd" containerName="payload-pre-processing" containerID="cri-o://d4f8efd9374f4043a3ba87713d580f6acd7ee4f4c7e619cb4be07268894cf755" gracePeriod=25 Oct 30 02:00:21.661231 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.661188 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" path="/var/lib/kubelet/pods/14084516-d9fb-4b98-8609-3f55b46688e4/volumes" Oct 30 02:00:21.662226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.662194 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" path="/var/lib/kubelet/pods/6d14078e-51dc-4bcf-9a9c-17ee7901be40/volumes" Oct 30 02:00:21.845313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.844401 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:21.858459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.858340 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:21.951003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.950968 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b822b327-db3b-459c-943a-39691f3875a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b822b327-db3b-459c-943a-39691f3875a1-maas-api-service-ca\") pod \"b822b327-db3b-459c-943a-39691f3875a1\" (UID: \"b822b327-db3b-459c-943a-39691f3875a1\") " Oct 30 02:00:21.951173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.951134 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b822b327-db3b-459c-943a-39691f3875a1-kube-api-access-xpnxb\" (UniqueName: \"kubernetes.io/projected/b822b327-db3b-459c-943a-39691f3875a1-kube-api-access-xpnxb\") pod \"b822b327-db3b-459c-943a-39691f3875a1\" (UID: \"b822b327-db3b-459c-943a-39691f3875a1\") " Oct 30 02:00:21.952246 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.952215 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b822b327-db3b-459c-943a-39691f3875a1-maas-api-service-ca" pod "b822b327-db3b-459c-943a-39691f3875a1" (UID: "b822b327-db3b-459c-943a-39691f3875a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.954743 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:21.954714 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b822b327-db3b-459c-943a-39691f3875a1-kube-api-access-xpnxb" pod "b822b327-db3b-459c-943a-39691f3875a1" (UID: "b822b327-db3b-459c-943a-39691f3875a1"). InnerVolumeSpecName "kube-api-access-xpnxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:22.053859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.052968 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0214eaff-b6f8-491f-9259-18431f48a390-kube-api-access-wznnt\" (UniqueName: \"kubernetes.io/projected/0214eaff-b6f8-491f-9259-18431f48a390-kube-api-access-wznnt\") pod \"0214eaff-b6f8-491f-9259-18431f48a390\" (UID: \"0214eaff-b6f8-491f-9259-18431f48a390\") " Oct 30 02:00:22.053859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.053041 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0214eaff-b6f8-491f-9259-18431f48a390-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0214eaff-b6f8-491f-9259-18431f48a390-maas-api-service-ca\") pod \"0214eaff-b6f8-491f-9259-18431f48a390\" (UID: \"0214eaff-b6f8-491f-9259-18431f48a390\") " Oct 30 02:00:22.053859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.053748 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b822b327-db3b-459c-943a-39691f3875a1-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.053859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.053769 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpnxb\" (UniqueName: \"kubernetes.io/projected/b822b327-db3b-459c-943a-39691f3875a1-kube-api-access-xpnxb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.058147 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.054971 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0214eaff-b6f8-491f-9259-18431f48a390-maas-api-service-ca" pod "0214eaff-b6f8-491f-9259-18431f48a390" (UID: "0214eaff-b6f8-491f-9259-18431f48a390"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.065386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.064948 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 02:00:22.065386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.065011 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 02:00:22.142109 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.090867 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:00:22.142109 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.090930 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.143636 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc"] Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144796 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144826 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144845 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144855 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144880 5035 cpu_manager.go:414] "containerMap: removing container" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144888 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144899 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144908 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144919 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b822b327-db3b-459c-943a-39691f3875a1" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144926 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b822b327-db3b-459c-943a-39691f3875a1" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144942 5035 cpu_manager.go:414] "containerMap: removing container" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144949 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144965 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0214eaff-b6f8-491f-9259-18431f48a390" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144972 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0214eaff-b6f8-491f-9259-18431f48a390" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144984 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b822b327-db3b-459c-943a-39691f3875a1" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.144991 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b822b327-db3b-459c-943a-39691f3875a1" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145007 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0214eaff-b6f8-491f-9259-18431f48a390" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145013 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0214eaff-b6f8-491f-9259-18431f48a390" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145243 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0214eaff-b6f8-491f-9259-18431f48a390" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145258 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b822b327-db3b-459c-943a-39691f3875a1" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145269 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0214eaff-b6f8-491f-9259-18431f48a390" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145286 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b822b327-db3b-459c-943a-39691f3875a1" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145297 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145312 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145323 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145335 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerName="cleanup" Oct 30 02:00:22.176648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.145344 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" containerName="cleanup" Oct 30 02:00:22.203615 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.187685 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0214eaff-b6f8-491f-9259-18431f48a390-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.266756 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.235026 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc"] Oct 30 02:00:22.266756 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.235170 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.315566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.288855 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-token\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.315566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.288916 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-credential-socket\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.315566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.288945 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-data\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.315566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.288973 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr4wq\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-kube-api-access-zr4wq\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.315566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.289011 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-envoy\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.315566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.289043 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-podinfo\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.315566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.289100 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504f0ee4-4923-41e6-aa6f-7c976333058a-istiod-ca-cert\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.315566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.289145 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-socket\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.315566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.289305 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-certs\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.387599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.387437 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0214eaff-b6f8-491f-9259-18431f48a390-kube-api-access-wznnt" pod "0214eaff-b6f8-491f-9259-18431f48a390" (UID: "0214eaff-b6f8-491f-9259-18431f48a390"). InnerVolumeSpecName "kube-api-access-wznnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521117 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504f0ee4-4923-41e6-aa6f-7c976333058a-istiod-ca-cert\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521190 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-socket\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521401 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-certs\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521575 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-token\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521608 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-credential-socket\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521640 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-data\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521672 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr4wq\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-kube-api-access-zr4wq\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521717 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-envoy\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521759 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-podinfo\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.524807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.521870 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznnt\" (UniqueName: \"kubernetes.io/projected/0214eaff-b6f8-491f-9259-18431f48a390-kube-api-access-wznnt\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.576261 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.559452 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504f0ee4-4923-41e6-aa6f-7c976333058a-istiod-ca-cert\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.576261 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.559935 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-socket\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.576261 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.560183 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-certs\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.576261 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.562471 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-data\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.576261 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.562726 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-credential-socket\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.576261 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.562956 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-38bc0d-openshift-default-dockercfg-glj89\"" Oct 30 02:00:22.638932 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.635003 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-envoy\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.638932 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.637866 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" podUID="24af78d5-514c-41c6-b972-ab3b298c69d1" containerName="payload-processing" containerID="cri-o://fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d" gracePeriod=25 Oct 30 02:00:22.650892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.642601 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-podinfo\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.650892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.648060 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr4wq\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-kube-api-access-zr4wq\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.652474 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.651736 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-token\") pod \"e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.660906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.659734 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" Oct 30 02:00:22.660906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.659823 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s" event={"ID":"b822b327-db3b-459c-943a-39691f3875a1","Type":"ContainerDied","Data":"e9264d47280a956a41866381f2ae9b251e8fc2c65e0fd6ed23a8106d2b700613"} Oct 30 02:00:22.660906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.660011 5035 scope.go:122] "RemoveContainer" containerID="8eb6ff8fea539c66eb573d48727c43509ae6d2d6aab1856f301c379656ce1093" Oct 30 02:00:22.715024 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.700043 5035 generic.go:356] "Generic (PLEG): container finished" podID="04b2d88d-c92d-49e6-84d6-e7267ae2b7fd" containerID="d4f8efd9374f4043a3ba87713d580f6acd7ee4f4c7e619cb4be07268894cf755" exitCode=0 Oct 30 02:00:22.715024 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.700352 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" event={"ID":"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd","Type":"ContainerDied","Data":"d4f8efd9374f4043a3ba87713d580f6acd7ee4f4c7e619cb4be07268894cf755"} Oct 30 02:00:22.730104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.729489 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" event={"ID":"0214eaff-b6f8-491f-9259-18431f48a390","Type":"ContainerDied","Data":"7895686aeab107be27e5e58bd5f0d09fcb7ccf4766359fef98ada90bb3bb1083"} Oct 30 02:00:22.731357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.729651 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm" Oct 30 02:00:22.733842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.733337 5035 generic.go:356] "Generic (PLEG): container finished" podID="6b9160fd-4164-484f-b5bb-88f8846934d4" containerID="ec3bce30a5a5cfa4f2bd7d20ff4b9a45daab62817deddeee648928d9d1c74dae" exitCode=0 Oct 30 02:00:22.733842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.733413 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" event={"ID":"6b9160fd-4164-484f-b5bb-88f8846934d4","Type":"ContainerDied","Data":"ec3bce30a5a5cfa4f2bd7d20ff4b9a45daab62817deddeee648928d9d1c74dae"} Oct 30 02:00:22.743209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.743179 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 02:00:22.748908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.748881 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:22.774829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.774754 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 02:00:22.829226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.829163 5035 scope.go:122] "RemoveContainer" containerID="c7cb60082b9204f9df77d57fb1efe4aeda9636ffe887c1ee7455f55c8f02593d" Oct 30 02:00:22.859854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.859820 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9160fd-4164-484f-b5bb-88f8846934d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9160fd-4164-484f-b5bb-88f8846934d4-plugins-config-volume\") pod \"6b9160fd-4164-484f-b5bb-88f8846934d4\" (UID: \"6b9160fd-4164-484f-b5bb-88f8846934d4\") " Oct 30 02:00:22.860909 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.860553 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9160fd-4164-484f-b5bb-88f8846934d4-plugins-config-volume" pod "6b9160fd-4164-484f-b5bb-88f8846934d4" (UID: "6b9160fd-4164-484f-b5bb-88f8846934d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.861082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.860982 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9160fd-4164-484f-b5bb-88f8846934d4-kube-api-access-z5br7\" (UniqueName: \"kubernetes.io/projected/6b9160fd-4164-484f-b5bb-88f8846934d4-kube-api-access-z5br7\") pod \"6b9160fd-4164-484f-b5bb-88f8846934d4\" (UID: \"6b9160fd-4164-484f-b5bb-88f8846934d4\") " Oct 30 02:00:22.862444 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.862415 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9160fd-4164-484f-b5bb-88f8846934d4-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.864926 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.864886 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9160fd-4164-484f-b5bb-88f8846934d4-kube-api-access-z5br7" pod "6b9160fd-4164-484f-b5bb-88f8846934d4" (UID: "6b9160fd-4164-484f-b5bb-88f8846934d4"). InnerVolumeSpecName "kube-api-access-z5br7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:22.865058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.865038 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s"] Oct 30 02:00:22.908332 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.908284 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888760-xvk2s"] Oct 30 02:00:22.927117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.927000 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm"] Oct 30 02:00:22.936146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.936092 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7542fa-29888760-gzpwm"] Oct 30 02:00:22.964570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.964145 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-kube-api-access-hcvcv\" (UniqueName: \"kubernetes.io/projected/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-kube-api-access-hcvcv\") pod \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\" (UID: \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\") " Oct 30 02:00:22.964570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.964285 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-plugins-config-volume\") pod \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\" (UID: \"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd\") " Oct 30 02:00:22.965758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.965689 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5br7\" (UniqueName: \"kubernetes.io/projected/6b9160fd-4164-484f-b5bb-88f8846934d4-kube-api-access-z5br7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.966462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:22.966393 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-plugins-config-volume" pod "04b2d88d-c92d-49e6-84d6-e7267ae2b7fd" (UID: "04b2d88d-c92d-49e6-84d6-e7267ae2b7fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.102672 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.067276 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.109037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.105639 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:00:23.109037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.105725 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 02:00:23.109037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.107772 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 02:00:23.109037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.107822 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 02:00:23.109037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.108256 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-kube-api-access-hcvcv" pod "04b2d88d-c92d-49e6-84d6-e7267ae2b7fd" (UID: "04b2d88d-c92d-49e6-84d6-e7267ae2b7fd"). InnerVolumeSpecName "kube-api-access-hcvcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.204978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.204944 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcvcv\" (UniqueName: \"kubernetes.io/projected/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd-kube-api-access-hcvcv\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.246445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.246397 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 02:00:23.313512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.313474 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc"] Oct 30 02:00:23.322751 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.322707 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:23.322930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.322804 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:23.323222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.323180 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:00:23.410890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.408681 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24af78d5-514c-41c6-b972-ab3b298c69d1-kube-api-access-gqw7z\" (UniqueName: \"kubernetes.io/projected/24af78d5-514c-41c6-b972-ab3b298c69d1-kube-api-access-gqw7z\") pod \"24af78d5-514c-41c6-b972-ab3b298c69d1\" (UID: \"24af78d5-514c-41c6-b972-ab3b298c69d1\") " Oct 30 02:00:23.410890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.409008 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24af78d5-514c-41c6-b972-ab3b298c69d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24af78d5-514c-41c6-b972-ab3b298c69d1-plugins-config-volume\") pod \"24af78d5-514c-41c6-b972-ab3b298c69d1\" (UID: \"24af78d5-514c-41c6-b972-ab3b298c69d1\") " Oct 30 02:00:23.410890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.409522 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24af78d5-514c-41c6-b972-ab3b298c69d1-plugins-config-volume" pod "24af78d5-514c-41c6-b972-ab3b298c69d1" (UID: "24af78d5-514c-41c6-b972-ab3b298c69d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.410890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.409851 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24af78d5-514c-41c6-b972-ab3b298c69d1-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.411512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.411274 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24af78d5-514c-41c6-b972-ab3b298c69d1-kube-api-access-gqw7z" pod "24af78d5-514c-41c6-b972-ab3b298c69d1" (UID: "24af78d5-514c-41c6-b972-ab3b298c69d1"). InnerVolumeSpecName "kube-api-access-gqw7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.512849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.512804 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqw7z\" (UniqueName: \"kubernetes.io/projected/24af78d5-514c-41c6-b972-ab3b298c69d1-kube-api-access-gqw7z\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.659598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.657826 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0214eaff-b6f8-491f-9259-18431f48a390" path="/var/lib/kubelet/pods/0214eaff-b6f8-491f-9259-18431f48a390/volumes" Oct 30 02:00:23.659598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.658542 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b822b327-db3b-459c-943a-39691f3875a1" path="/var/lib/kubelet/pods/b822b327-db3b-459c-943a-39691f3875a1/volumes" Oct 30 02:00:23.755262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.755230 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" Oct 30 02:00:23.755957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.755251 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss" event={"ID":"6b9160fd-4164-484f-b5bb-88f8846934d4","Type":"ContainerDied","Data":"e80c3d67b39d21051eefda1d5f23b0f60ddc2a3aa683afb6bbcb5c8d9118b842"} Oct 30 02:00:23.756086 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.756016 5035 scope.go:122] "RemoveContainer" containerID="ec3bce30a5a5cfa4f2bd7d20ff4b9a45daab62817deddeee648928d9d1c74dae" Oct 30 02:00:23.759471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.759323 5035 generic.go:356] "Generic (PLEG): container finished" podID="24af78d5-514c-41c6-b972-ab3b298c69d1" containerID="fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d" exitCode=0 Oct 30 02:00:23.759471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.759437 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" Oct 30 02:00:23.759639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.759609 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" event={"ID":"24af78d5-514c-41c6-b972-ab3b298c69d1","Type":"ContainerDied","Data":"fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d"} Oct 30 02:00:23.759756 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.759675 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k" event={"ID":"24af78d5-514c-41c6-b972-ab3b298c69d1","Type":"ContainerDied","Data":"4d4ae4612e4e93cbb5970486e4cf5e104e1ebf3a04bb648251956540883485f6"} Oct 30 02:00:23.764120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.764085 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" Oct 30 02:00:23.764120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.764099 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l" event={"ID":"04b2d88d-c92d-49e6-84d6-e7267ae2b7fd","Type":"ContainerDied","Data":"44f2a1ba34ef27c0e5d44035759452555161dd903f3be03f71cbe9da23687eff"} Oct 30 02:00:23.769761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.769730 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" event={"ID":"504f0ee4-4923-41e6-aa6f-7c976333058a","Type":"ContainerStarted","Data":"d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3"} Oct 30 02:00:23.769911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.769778 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" event={"ID":"504f0ee4-4923-41e6-aa6f-7c976333058a","Type":"ContainerStarted","Data":"e91f08fc8b5a3421f4d02fd4b96a346ea2b0b65c897ce342317084520eb1f1ab"} Oct 30 02:00:23.777859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.777830 5035 scope.go:122] "RemoveContainer" containerID="fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d" Oct 30 02:00:23.790941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.790905 5035 scope.go:122] "RemoveContainer" containerID="fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d" Oct 30 02:00:23.791301 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:23.791280 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d\": container with ID starting with fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d not found: ID does not exist" containerID="fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d" Oct 30 02:00:23.791401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.791310 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d"} err="failed to get container status \"fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d\": rpc error: code = NotFound desc = could not find container \"fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d\": container with ID starting with fd7c42360acbafd610c452e0e03f10689db2ba047489592d66194dae7cb3cd6d not found: ID does not exist" Oct 30 02:00:23.791401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.791329 5035 scope.go:122] "RemoveContainer" containerID="d4f8efd9374f4043a3ba87713d580f6acd7ee4f4c7e619cb4be07268894cf755" Oct 30 02:00:23.824687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.824619 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" podStartSLOduration=2.824595509 podStartE2EDuration="2.824595509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:23.822687251 +0000 UTC m=+1902.325619463" watchObservedRunningTime="2026-10-30 02:00:23.824595509 +0000 UTC m=+1902.327527747" Oct 30 02:00:23.862969 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.862933 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l"] Oct 30 02:00:23.874328 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.874292 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6ba1d9-7c9cdf8fc9-4874l"] Oct 30 02:00:23.912600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.912463 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss"] Oct 30 02:00:23.921002 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.920962 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6ba1d9-765d5dfb88-4djss"] Oct 30 02:00:23.959361 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.959322 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k"] Oct 30 02:00:23.964978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:23.964935 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c90f5-5c75d4fcc6-zft8k"] Oct 30 02:00:24.059161 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.059124 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:24.059446 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.059426 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:24.060267 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.060248 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:00:24.095635 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.095603 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:24.096014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.095989 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:24.097050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.097030 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:00:24.749610 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.749574 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:24.755359 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.755328 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:24.779691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.779655 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:24.780716 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:24.780692 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:00:25.092139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.092042 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr"] Oct 30 02:00:25.093488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.093452 5035 cpu_manager.go:414] "containerMap: removing container" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" containerName="cleanup" Oct 30 02:00:25.093693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.093676 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14084516-d9fb-4b98-8609-3f55b46688e4" containerName="cleanup" Oct 30 02:00:25.093795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.093784 5035 cpu_manager.go:414] "containerMap: removing container" podUID="04b2d88d-c92d-49e6-84d6-e7267ae2b7fd" containerName="payload-pre-processing" Oct 30 02:00:25.093897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.093885 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b2d88d-c92d-49e6-84d6-e7267ae2b7fd" containerName="payload-pre-processing" Oct 30 02:00:25.093978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.093968 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b822b327-db3b-459c-943a-39691f3875a1" containerName="cleanup" Oct 30 02:00:25.094060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.094049 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b822b327-db3b-459c-943a-39691f3875a1" containerName="cleanup" Oct 30 02:00:25.094135 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.094126 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0214eaff-b6f8-491f-9259-18431f48a390" containerName="cleanup" Oct 30 02:00:25.094207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.094198 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0214eaff-b6f8-491f-9259-18431f48a390" containerName="cleanup" Oct 30 02:00:25.094317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.094308 5035 cpu_manager.go:414] "containerMap: removing container" podUID="24af78d5-514c-41c6-b972-ab3b298c69d1" containerName="payload-processing" Oct 30 02:00:25.094393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.094383 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24af78d5-514c-41c6-b972-ab3b298c69d1" containerName="payload-processing" Oct 30 02:00:25.094494 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.094484 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6b9160fd-4164-484f-b5bb-88f8846934d4" containerName="payload-processing" Oct 30 02:00:25.094654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.094635 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9160fd-4164-484f-b5bb-88f8846934d4" containerName="payload-processing" Oct 30 02:00:25.095010 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.094986 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b2d88d-c92d-49e6-84d6-e7267ae2b7fd" containerName="payload-pre-processing" Oct 30 02:00:25.095137 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.095125 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0214eaff-b6f8-491f-9259-18431f48a390" containerName="cleanup" Oct 30 02:00:25.095238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.095226 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="24af78d5-514c-41c6-b972-ab3b298c69d1" containerName="payload-processing" Oct 30 02:00:25.095333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.095323 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b822b327-db3b-459c-943a-39691f3875a1" containerName="cleanup" Oct 30 02:00:25.095430 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.095419 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9160fd-4164-484f-b5bb-88f8846934d4" containerName="payload-processing" Oct 30 02:00:25.095544 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.095517 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d14078e-51dc-4bcf-9a9c-17ee7901be40" containerName="cleanup" Oct 30 02:00:25.114805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.114766 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr"] Oct 30 02:00:25.114997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.114898 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:25.134559 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.133866 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-145336\"" Oct 30 02:00:25.134559 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.134154 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-145336-dockercfg-q44ql\"" Oct 30 02:00:25.250579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.250519 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpx5k\" (UniqueName: \"kubernetes.io/projected/e3aeea64-7a64-4298-acff-44762f5835d4-kube-api-access-lpx5k\") pod \"payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr\" (UID: \"e3aeea64-7a64-4298-acff-44762f5835d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:25.250774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.250677 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aeea64-7a64-4298-acff-44762f5835d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr\" (UID: \"e3aeea64-7a64-4298-acff-44762f5835d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:25.352453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.352342 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpx5k\" (UniqueName: \"kubernetes.io/projected/e3aeea64-7a64-4298-acff-44762f5835d4-kube-api-access-lpx5k\") pod \"payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr\" (UID: \"e3aeea64-7a64-4298-acff-44762f5835d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:25.352453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.352435 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aeea64-7a64-4298-acff-44762f5835d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr\" (UID: \"e3aeea64-7a64-4298-acff-44762f5835d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:25.353207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.353183 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aeea64-7a64-4298-acff-44762f5835d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr\" (UID: \"e3aeea64-7a64-4298-acff-44762f5835d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:25.377415 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.377375 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpx5k\" (UniqueName: \"kubernetes.io/projected/e3aeea64-7a64-4298-acff-44762f5835d4-kube-api-access-lpx5k\") pod \"payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr\" (UID: \"e3aeea64-7a64-4298-acff-44762f5835d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:25.467932 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.467880 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:25.648891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.648807 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b2d88d-c92d-49e6-84d6-e7267ae2b7fd" path="/var/lib/kubelet/pods/04b2d88d-c92d-49e6-84d6-e7267ae2b7fd/volumes" Oct 30 02:00:25.649434 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.649414 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24af78d5-514c-41c6-b972-ab3b298c69d1" path="/var/lib/kubelet/pods/24af78d5-514c-41c6-b972-ab3b298c69d1/volumes" Oct 30 02:00:25.649943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.649928 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9160fd-4164-484f-b5bb-88f8846934d4" path="/var/lib/kubelet/pods/6b9160fd-4164-484f-b5bb-88f8846934d4/volumes" Oct 30 02:00:25.799385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:25.799338 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr"] Oct 30 02:00:26.850309 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:26.850267 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" event={"ID":"e3aeea64-7a64-4298-acff-44762f5835d4","Type":"ContainerStarted","Data":"c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003"} Oct 30 02:00:26.850875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:26.850335 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" event={"ID":"e3aeea64-7a64-4298-acff-44762f5835d4","Type":"ContainerStarted","Data":"3684b4c7560e951c4ed02c0174b9940dff44ef56e8eb792bfc7922ff2825a47c"} Oct 30 02:00:26.850875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:26.850369 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:26.899244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:26.899185 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" podStartSLOduration=1.899169406 podStartE2EDuration="1.899169406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:26.897938672 +0000 UTC m=+1905.400870893" watchObservedRunningTime="2026-10-30 02:00:26.899169406 +0000 UTC m=+1905.402101723" Oct 30 02:00:28.304671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.304631 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925"] Oct 30 02:00:28.309650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.309617 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:28.329583 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.329544 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925"] Oct 30 02:00:28.338092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.338049 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-38bc0d\"" Oct 30 02:00:28.338275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.338131 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-38bc0d-dockercfg-frkj5\"" Oct 30 02:00:28.475560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.471764 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925\" (UID: \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:28.475560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.471830 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v28bh\" (UniqueName: \"kubernetes.io/projected/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-kube-api-access-v28bh\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925\" (UID: \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:28.575082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.574905 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925\" (UID: \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:28.575082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.574981 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v28bh\" (UniqueName: \"kubernetes.io/projected/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-kube-api-access-v28bh\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925\" (UID: \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:28.576350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.576316 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925\" (UID: \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:28.607095 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.607009 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v28bh\" (UniqueName: \"kubernetes.io/projected/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-kube-api-access-v28bh\") pod \"payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925\" (UID: \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:28.642467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.641568 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:28.996115 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:28.996065 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7654c0_f569_4dbd_a6e4_52cfa6cc8180.slice/crio-159ce8661eb043ff501e62e07c3f464e7a8570467ae1cad00fcc0ada46af5ff5 WatchSource:0}: Error finding container 159ce8661eb043ff501e62e07c3f464e7a8570467ae1cad00fcc0ada46af5ff5: Status 404 returned error can't find the container with id 159ce8661eb043ff501e62e07c3f464e7a8570467ae1cad00fcc0ada46af5ff5 Oct 30 02:00:28.997063 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:28.997034 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925"] Oct 30 02:00:29.946433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:29.946389 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" event={"ID":"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180","Type":"ContainerStarted","Data":"c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363"} Oct 30 02:00:29.946958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:29.946455 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" event={"ID":"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180","Type":"ContainerStarted","Data":"159ce8661eb043ff501e62e07c3f464e7a8570467ae1cad00fcc0ada46af5ff5"} Oct 30 02:00:29.946958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:29.946479 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:29.999973 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:29.999913 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" podStartSLOduration=1.9998867649999998 podStartE2EDuration="1.999886765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:29.996321319 +0000 UTC m=+1908.499253540" watchObservedRunningTime="2026-10-30 02:00:29.999886765 +0000 UTC m=+1908.502819056" Oct 30 02:00:37.864490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:37.864456 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:00:40.958440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:40.958402 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:00:42.883183 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:42.883143 5035 scope.go:122] "RemoveContainer" containerID="c77b5d0d286d5bdc6a79a82aa3d8566224b3a2a13bba2951e3e9beb633041a13" Oct 30 02:00:42.894611 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:42.894584 5035 scope.go:122] "RemoveContainer" containerID="89d4cf7bff88e54f80e51d9fa2ae0a55c37e454f08598e18bb6903d1a42bcf9b" Oct 30 02:00:42.906109 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:42.906076 5035 scope.go:122] "RemoveContainer" containerID="5e790a328ae4d9e08612cf0506a6a7e108236aef6ac4348eb715ab6a2310718a" Oct 30 02:00:42.917347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:42.917320 5035 scope.go:122] "RemoveContainer" containerID="a7e5bd42f6964847b701bcd96fe497a8cfb817f7789ad6787d6601f5048b41b5" Oct 30 02:00:42.928641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:42.928615 5035 scope.go:122] "RemoveContainer" containerID="73c1dd092aa52f0b9a477b1740d14614e55725bb78a14f355098109c57ad5ff3" Oct 30 02:00:47.631542 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:00:47.631485 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:00:50.254078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.254032 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8"] Oct 30 02:00:50.260961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.260933 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.277604 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.276956 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-145336-serving-cert\"" Oct 30 02:00:50.281367 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.281338 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8"] Oct 30 02:00:50.323835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.323796 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6drj\" (UniqueName: \"kubernetes.io/projected/d7fb44dd-ba03-4128-967d-bfaf2fc21090-kube-api-access-b6drj\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.324100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.324078 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-metrics-certs\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.324149 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.324116 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7fb44dd-ba03-4128-967d-bfaf2fc21090-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.324214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.324203 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-maas-api-tls\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.425818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.425772 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-metrics-certs\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.426047 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.425832 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7fb44dd-ba03-4128-967d-bfaf2fc21090-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.426047 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.426030 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-maas-api-tls\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.426170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.426138 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6drj\" (UniqueName: \"kubernetes.io/projected/d7fb44dd-ba03-4128-967d-bfaf2fc21090-kube-api-access-b6drj\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.426663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.426635 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7fb44dd-ba03-4128-967d-bfaf2fc21090-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.428748 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.428717 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-maas-api-tls\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.428877 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.428717 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-metrics-certs\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.468061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.468024 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6drj\" (UniqueName: \"kubernetes.io/projected/d7fb44dd-ba03-4128-967d-bfaf2fc21090-kube-api-access-b6drj\") pod \"maas-api-e2e-ipp-b-145336-786d75f756-8jsp8\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.583211 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.583111 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:50.910499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:50.910465 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8"] Oct 30 02:00:50.911563 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:50.911519 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fb44dd_ba03_4128_967d_bfaf2fc21090.slice/crio-cdd64a2eda95ba13e4330af356ca31e514d5637acb5ce5afe76b9b73e47b30dc WatchSource:0}: Error finding container cdd64a2eda95ba13e4330af356ca31e514d5637acb5ce5afe76b9b73e47b30dc: Status 404 returned error can't find the container with id cdd64a2eda95ba13e4330af356ca31e514d5637acb5ce5afe76b9b73e47b30dc Oct 30 02:00:51.149501 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:51.149460 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" event={"ID":"d7fb44dd-ba03-4128-967d-bfaf2fc21090","Type":"ContainerStarted","Data":"cdd64a2eda95ba13e4330af356ca31e514d5637acb5ce5afe76b9b73e47b30dc"} Oct 30 02:00:52.159137 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:52.159089 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" event={"ID":"d7fb44dd-ba03-4128-967d-bfaf2fc21090","Type":"ContainerStarted","Data":"67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26"} Oct 30 02:00:52.159594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:52.159166 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:00:52.207129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:52.207071 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" podStartSLOduration=1.794294357 podStartE2EDuration="2.20705582s" totalImagesPullingTime="412.761463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:52.20371743 +0000 UTC m=+1930.706649651" watchObservedRunningTime="2026-10-30 02:00:52.20705582 +0000 UTC m=+1930.709988040" Oct 30 02:00:53.850093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:53.850008 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz"] Oct 30 02:00:53.854730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:53.854704 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:53.901425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:53.901384 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-38bc0d-serving-cert\"" Oct 30 02:00:53.906108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:53.906075 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz"] Oct 30 02:00:54.006129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.006085 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf97867-abb8-41cc-8302-a6e9fce75769-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.006129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.006133 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-maas-api-tls\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.006412 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.006156 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-metrics-certs\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.006412 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.006349 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p72xb\" (UniqueName: \"kubernetes.io/projected/faf97867-abb8-41cc-8302-a6e9fce75769-kube-api-access-p72xb\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.110612 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.110455 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf97867-abb8-41cc-8302-a6e9fce75769-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.110612 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.110520 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-maas-api-tls\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.110880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.110842 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-metrics-certs\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.111552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.111179 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p72xb\" (UniqueName: \"kubernetes.io/projected/faf97867-abb8-41cc-8302-a6e9fce75769-kube-api-access-p72xb\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.111552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.111380 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf97867-abb8-41cc-8302-a6e9fce75769-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.114448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.114423 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-maas-api-tls\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.114658 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.114636 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-metrics-certs\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.137228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.137195 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p72xb\" (UniqueName: \"kubernetes.io/projected/faf97867-abb8-41cc-8302-a6e9fce75769-kube-api-access-p72xb\") pod \"maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.208077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.208040 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:54.544187 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:54.544159 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz"] Oct 30 02:00:54.545488 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:00:54.545448 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf97867_abb8_41cc_8302_a6e9fce75769.slice/crio-1f36a08d89066cc4da5271949b0929b1c5f700009920e0086e23a430b352365a WatchSource:0}: Error finding container 1f36a08d89066cc4da5271949b0929b1c5f700009920e0086e23a430b352365a: Status 404 returned error can't find the container with id 1f36a08d89066cc4da5271949b0929b1c5f700009920e0086e23a430b352365a Oct 30 02:00:55.209883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:55.209843 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" event={"ID":"faf97867-abb8-41cc-8302-a6e9fce75769","Type":"ContainerStarted","Data":"1f36a08d89066cc4da5271949b0929b1c5f700009920e0086e23a430b352365a"} Oct 30 02:00:56.218622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:56.218580 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" event={"ID":"faf97867-abb8-41cc-8302-a6e9fce75769","Type":"ContainerStarted","Data":"cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850"} Oct 30 02:00:56.219043 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:56.218733 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:00:56.286031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:56.285950 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" podStartSLOduration=2.776005762 podStartE2EDuration="3.285924984s" totalImagesPullingTime="509.919222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:56.263609962 +0000 UTC m=+1934.766542195" watchObservedRunningTime="2026-10-30 02:00:56.285924984 +0000 UTC m=+1934.788857208" Oct 30 02:00:58.193633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:00:58.193599 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:01:02.234439 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:02.234405 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:01:03.284381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:03.284342 5035 generic.go:356] "Generic (PLEG): container finished" podID="3879878f-0211-4689-b111-62aead00bccf" containerID="cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684" exitCode=0 Oct 30 02:01:03.284896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:03.284414 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" event={"ID":"3879878f-0211-4689-b111-62aead00bccf","Type":"ContainerDied","Data":"cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684"} Oct 30 02:01:03.284896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:03.284880 5035 scope.go:122] "RemoveContainer" containerID="cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684" Oct 30 02:01:03.286636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:03.286607 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:03.286713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:03.286678 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:03.286755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:03.286711 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:03.875971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:03.875930 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:01:04.305454 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:04.305419 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" event={"ID":"3879878f-0211-4689-b111-62aead00bccf","Type":"ContainerStarted","Data":"52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0"} Oct 30 02:01:05.048977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:05.048931 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:01:05.050506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:05.050475 5035 patch_prober.go:29] interesting pod/e2e-api-b-356082-openshift-default-7dd6879f-cf82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:01:05.050602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:05.050561 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 02:01:06.052079 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:06.052041 5035 patch_prober.go:29] interesting pod/e2e-api-b-356082-openshift-default-7dd6879f-cf82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:01:06.052626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:06.052106 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 02:01:07.049714 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:07.049671 5035 patch_prober.go:29] interesting pod/e2e-api-b-356082-openshift-default-7dd6879f-cf82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:01:07.049936 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:07.049739 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 02:01:08.049173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:08.049123 5035 patch_prober.go:29] interesting pod/e2e-api-b-356082-openshift-default-7dd6879f-cf82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:01:08.049757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:08.049192 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 02:01:09.049849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:09.049808 5035 patch_prober.go:29] interesting pod/e2e-api-b-356082-openshift-default-7dd6879f-cf82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:01:09.050271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:09.049863 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 02:01:10.049015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:10.048979 5035 patch_prober.go:29] interesting pod/e2e-api-b-356082-openshift-default-7dd6879f-cf82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:01:10.049359 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:10.049321 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 02:01:10.056067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:10.056029 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87"] Oct 30 02:01:10.056583 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:10.056280 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerName="istio-proxy" containerID="cri-o://245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c" gracePeriod=30 Oct 30 02:01:10.108795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:10.108761 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv"] Oct 30 02:01:10.109190 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:10.109166 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" containerID="cri-o://b42c92fc1b1d26f61912b82e30612d97e6b7f9d235d489df9b43deb1cd5d7ec0" gracePeriod=30 Oct 30 02:01:11.054863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:11.054826 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:01:11.055182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:11.055144 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:01:11.056130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:11.056107 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:01:12.106586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.106548 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h"] Oct 30 02:01:12.110918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.110892 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.143060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.143013 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-95daa2-openshift-default-dockercfg-qttt5\"" Oct 30 02:01:12.162382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.162342 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h"] Oct 30 02:01:12.277553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.277188 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-socket\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.277553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.277289 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-podinfo\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.277553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.277351 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-credential-socket\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.277553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.277381 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-data\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.277553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.277415 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-certs\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.277553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.277487 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7m4\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-kube-api-access-hq7m4\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.278058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.277618 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-envoy\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.278058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.277725 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-token\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.278058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.277751 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae724a0-57d2-4116-9537-31e7aa2aea87-istiod-ca-cert\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.392626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.379712 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-envoy\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.404960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.398028 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-token\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.405513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.405441 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae724a0-57d2-4116-9537-31e7aa2aea87-istiod-ca-cert\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.420565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.416863 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-socket\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.462755 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-podinfo\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.462852 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-credential-socket\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.462880 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-data\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.462917 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-certs\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.463002 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7m4\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-kube-api-access-hq7m4\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.440555 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-token\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.463487 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-credential-socket\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.416640 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae724a0-57d2-4116-9537-31e7aa2aea87-istiod-ca-cert\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.421932 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-socket\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.445141 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-envoy\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.463931 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-certs\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.465878 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-data\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.467545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.466856 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-podinfo\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.496960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.496904 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7m4\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-kube-api-access-hq7m4\") pod \"e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:12.728325 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:12.728283 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:13.094881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:13.094827 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h"] Oct 30 02:01:13.096163 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:01:13.096129 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae724a0_57d2_4116_9537_31e7aa2aea87.slice/crio-25bbdfea74500f84706dddd0cac89d8c9ac818a5c96a5a08f839dbd04b7830b5 WatchSource:0}: Error finding container 25bbdfea74500f84706dddd0cac89d8c9ac818a5c96a5a08f839dbd04b7830b5: Status 404 returned error can't find the container with id 25bbdfea74500f84706dddd0cac89d8c9ac818a5c96a5a08f839dbd04b7830b5 Oct 30 02:01:13.099327 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:13.099289 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:13.099444 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:13.099363 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:13.099444 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:13.099403 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:13.470679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:13.470635 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" event={"ID":"2ae724a0-57d2-4116-9537-31e7aa2aea87","Type":"ContainerStarted","Data":"b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9"} Oct 30 02:01:13.471111 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:13.470700 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" event={"ID":"2ae724a0-57d2-4116-9537-31e7aa2aea87","Type":"ContainerStarted","Data":"25bbdfea74500f84706dddd0cac89d8c9ac818a5c96a5a08f839dbd04b7830b5"} Oct 30 02:01:13.548836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:13.548781 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" podStartSLOduration=1.548763692 podStartE2EDuration="1.548763692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:13.531469615 +0000 UTC m=+1952.034401836" watchObservedRunningTime="2026-10-30 02:01:13.548763692 +0000 UTC m=+1952.051695911" Oct 30 02:01:13.729423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:13.729328 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:14.175421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:14.175328 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb"] Oct 30 02:01:14.213684 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:14.213646 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w"] Oct 30 02:01:14.736453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:14.736414 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:15.448355 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.447634 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 02:01:15.492664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.492622 5035 generic.go:356] "Generic (PLEG): container finished" podID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerID="b42c92fc1b1d26f61912b82e30612d97e6b7f9d235d489df9b43deb1cd5d7ec0" exitCode=0 Oct 30 02:01:15.493722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.492704 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" event={"ID":"2c118c25-2881-4ec9-a079-fdae4c67f8fd","Type":"ContainerDied","Data":"b42c92fc1b1d26f61912b82e30612d97e6b7f9d235d489df9b43deb1cd5d7ec0"} Oct 30 02:01:15.493722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.492777 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" event={"ID":"2c118c25-2881-4ec9-a079-fdae4c67f8fd","Type":"ContainerDied","Data":"8bcf54bc3849a6a337eb61e75f00ef2ba27d5322dd0fb42a8589b34c403254bd"} Oct 30 02:01:15.493722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.492804 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bcf54bc3849a6a337eb61e75f00ef2ba27d5322dd0fb42a8589b34c403254bd" Oct 30 02:01:15.495039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.495007 5035 generic.go:356] "Generic (PLEG): container finished" podID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerID="245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c" exitCode=0 Oct 30 02:01:15.495188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.495171 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" Oct 30 02:01:15.495452 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.495422 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" event={"ID":"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45","Type":"ContainerDied","Data":"245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c"} Oct 30 02:01:15.495614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.495499 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87" event={"ID":"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45","Type":"ContainerDied","Data":"1a2cf063787f74644e976d23d05e6383e5833206da9d80dddaf9246ef9d937bc"} Oct 30 02:01:15.495614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.495596 5035 scope.go:122] "RemoveContainer" containerID="245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c" Oct 30 02:01:15.496736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.496505 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:15.497923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.497904 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:01:15.509904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.509870 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 02:01:15.516617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.516561 5035 scope.go:122] "RemoveContainer" containerID="245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c" Oct 30 02:01:15.517099 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:01:15.517063 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c\": container with ID starting with 245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c not found: ID does not exist" containerID="245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c" Oct 30 02:01:15.517235 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.517109 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c"} err="failed to get container status \"245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c\": rpc error: code = NotFound desc = could not find container \"245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c\": container with ID starting with 245e22a3549016e52c2904141899506b2de3980697cba75f8e43dae492672e7c not found: ID does not exist" Oct 30 02:01:15.585607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.585558 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-token\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-token\") pod \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " Oct 30 02:01:15.585798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.585656 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-envoy\") pod \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " Oct 30 02:01:15.585798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.585719 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-certs\") pod \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " Oct 30 02:01:15.585798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.585786 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-socket\") pod \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " Oct 30 02:01:15.585911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.585822 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-podinfo\") pod \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " Oct 30 02:01:15.585911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.585887 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-token\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-token\") pod \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " Oct 30 02:01:15.586074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.585948 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-envoy\") pod \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " Oct 30 02:01:15.586074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586005 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-kube-api-access-d8hs9\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-kube-api-access-d8hs9\") pod \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " Oct 30 02:01:15.586215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586113 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-podinfo\") pod \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " Oct 30 02:01:15.586215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586123 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-certs" pod "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" (UID: "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.586215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586163 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-data\") pod \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " Oct 30 02:01:15.586215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586209 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-kube-api-access-vvkfs\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-kube-api-access-vvkfs\") pod \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " Oct 30 02:01:15.586426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586256 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-certs\") pod \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " Oct 30 02:01:15.586426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586314 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-credential-socket\") pod \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " Oct 30 02:01:15.586426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586342 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-socket\") pod \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " Oct 30 02:01:15.586426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586387 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-data\") pod \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " Oct 30 02:01:15.586426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586416 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istiod-ca-cert\") pod \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " Oct 30 02:01:15.586753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586472 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-credential-socket\") pod \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\" (UID: \"0d0b842e-01f5-4bb9-9dcb-efc9577a6e45\") " Oct 30 02:01:15.586753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586546 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istiod-ca-cert\") pod \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\" (UID: \"2c118c25-2881-4ec9-a079-fdae4c67f8fd\") " Oct 30 02:01:15.587324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.586986 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-data" pod "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" (UID: "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.587732 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.587704 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-socket" pod "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" (UID: "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.588303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.588174 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-data" pod "2c118c25-2881-4ec9-a079-fdae4c67f8fd" (UID: "2c118c25-2881-4ec9-a079-fdae4c67f8fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.588303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.588251 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.588303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.588277 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.588303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.588291 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.588603 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.588481 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istiod-ca-cert" pod "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" (UID: "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:15.588844 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.588655 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-certs" pod "2c118c25-2881-4ec9-a079-fdae4c67f8fd" (UID: "2c118c25-2881-4ec9-a079-fdae4c67f8fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.588916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.588881 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-credential-socket" pod "2c118c25-2881-4ec9-a079-fdae4c67f8fd" (UID: "2c118c25-2881-4ec9-a079-fdae4c67f8fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.588957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.588917 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-credential-socket" pod "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" (UID: "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.589097 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.589038 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-token" pod "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" (UID: "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.589184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.589129 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-socket" pod "2c118c25-2881-4ec9-a079-fdae4c67f8fd" (UID: "2c118c25-2881-4ec9-a079-fdae4c67f8fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.589453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.589417 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istiod-ca-cert" pod "2c118c25-2881-4ec9-a079-fdae4c67f8fd" (UID: "2c118c25-2881-4ec9-a079-fdae4c67f8fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:15.589677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.589632 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-token" pod "2c118c25-2881-4ec9-a079-fdae4c67f8fd" (UID: "2c118c25-2881-4ec9-a079-fdae4c67f8fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.589911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.589886 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-envoy" pod "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" (UID: "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.590771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.590743 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-envoy" pod "2c118c25-2881-4ec9-a079-fdae4c67f8fd" (UID: "2c118c25-2881-4ec9-a079-fdae4c67f8fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.591597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.591560 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-kube-api-access-d8hs9" pod "2c118c25-2881-4ec9-a079-fdae4c67f8fd" (UID: "2c118c25-2881-4ec9-a079-fdae4c67f8fd"). InnerVolumeSpecName "kube-api-access-d8hs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.592570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.592519 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-podinfo" pod "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" (UID: "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:15.592845 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.592817 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-kube-api-access-vvkfs" pod "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" (UID: "0d0b842e-01f5-4bb9-9dcb-efc9577a6e45"). InnerVolumeSpecName "kube-api-access-vvkfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.593762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.593740 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-podinfo" pod "2c118c25-2881-4ec9-a079-fdae4c67f8fd" (UID: "2c118c25-2881-4ec9-a079-fdae4c67f8fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690068 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690112 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690126 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690139 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690155 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690168 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690183 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690197 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690209 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690224 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690236 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8hs9\" (UniqueName: \"kubernetes.io/projected/2c118c25-2881-4ec9-a079-fdae4c67f8fd-kube-api-access-d8hs9\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690250 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c118c25-2881-4ec9-a079-fdae4c67f8fd-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690264 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvkfs\" (UniqueName: \"kubernetes.io/projected/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45-kube-api-access-vvkfs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690279 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.693579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.690292 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c118c25-2881-4ec9-a079-fdae4c67f8fd-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.845694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.845653 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87"] Oct 30 02:01:15.852192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:15.852152 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-xnn87"] Oct 30 02:01:16.504687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:16.504649 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv" Oct 30 02:01:16.565164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:16.565115 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv"] Oct 30 02:01:16.581064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:16.581026 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-4wrpv"] Oct 30 02:01:17.636186 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:01:17.636139 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:01:17.646353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:17.646312 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" path="/var/lib/kubelet/pods/0d0b842e-01f5-4bb9-9dcb-efc9577a6e45/volumes" Oct 30 02:01:17.646971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:17.646952 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" path="/var/lib/kubelet/pods/2c118c25-2881-4ec9-a079-fdae4c67f8fd/volumes" Oct 30 02:01:18.521269 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:18.521233 5035 generic.go:356] "Generic (PLEG): container finished" podID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerID="d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3" exitCode=0 Oct 30 02:01:18.521482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:18.521300 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" event={"ID":"504f0ee4-4923-41e6-aa6f-7c976333058a","Type":"ContainerDied","Data":"d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3"} Oct 30 02:01:18.521810 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:18.521794 5035 scope.go:122] "RemoveContainer" containerID="d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3" Oct 30 02:01:18.523705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:18.523675 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:18.523801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:18.523760 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:18.523861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:18.523797 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:19.180552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.178859 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" podUID="c0082d45-3b3e-446f-829e-1dcb1eb71676" containerName="payload-processing" containerID="cri-o://15fafa42409166c76b7e9e6b91383f846c5e4e1d5015e4ba239c3fca8c6cc51a" gracePeriod=25 Oct 30 02:01:19.214193 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.214137 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" podUID="1a06db84-72df-4a11-8164-0353070babca" containerName="payload-pre-processing" containerID="cri-o://728dda8fbe7b8cdfa56c233379582ce533db2aa9cf57b01f29936782f3e5b1e6" gracePeriod=25 Oct 30 02:01:19.537077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.537044 5035 generic.go:356] "Generic (PLEG): container finished" podID="c0082d45-3b3e-446f-829e-1dcb1eb71676" containerID="15fafa42409166c76b7e9e6b91383f846c5e4e1d5015e4ba239c3fca8c6cc51a" exitCode=0 Oct 30 02:01:19.537275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.537149 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" event={"ID":"c0082d45-3b3e-446f-829e-1dcb1eb71676","Type":"ContainerDied","Data":"15fafa42409166c76b7e9e6b91383f846c5e4e1d5015e4ba239c3fca8c6cc51a"} Oct 30 02:01:19.537275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.537201 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" event={"ID":"c0082d45-3b3e-446f-829e-1dcb1eb71676","Type":"ContainerDied","Data":"3dd42969c9727b29710a059f78b6afb56fd67531764bbfd29621e50dafef13ad"} Oct 30 02:01:19.537275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.537227 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dd42969c9727b29710a059f78b6afb56fd67531764bbfd29621e50dafef13ad" Oct 30 02:01:19.539671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.539631 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" event={"ID":"504f0ee4-4923-41e6-aa6f-7c976333058a","Type":"ContainerStarted","Data":"fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf"} Oct 30 02:01:19.541999 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.541962 5035 generic.go:356] "Generic (PLEG): container finished" podID="1a06db84-72df-4a11-8164-0353070babca" containerID="728dda8fbe7b8cdfa56c233379582ce533db2aa9cf57b01f29936782f3e5b1e6" exitCode=0 Oct 30 02:01:19.542260 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.542238 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" event={"ID":"1a06db84-72df-4a11-8164-0353070babca","Type":"ContainerDied","Data":"728dda8fbe7b8cdfa56c233379582ce533db2aa9cf57b01f29936782f3e5b1e6"} Oct 30 02:01:19.560450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.560051 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 02:01:19.653477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.653451 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 02:01:19.664100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.664063 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0082d45-3b3e-446f-829e-1dcb1eb71676-kube-api-access-nrmf2\" (UniqueName: \"kubernetes.io/projected/c0082d45-3b3e-446f-829e-1dcb1eb71676-kube-api-access-nrmf2\") pod \"c0082d45-3b3e-446f-829e-1dcb1eb71676\" (UID: \"c0082d45-3b3e-446f-829e-1dcb1eb71676\") " Oct 30 02:01:19.664290 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.664228 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0082d45-3b3e-446f-829e-1dcb1eb71676-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0082d45-3b3e-446f-829e-1dcb1eb71676-plugins-config-volume\") pod \"c0082d45-3b3e-446f-829e-1dcb1eb71676\" (UID: \"c0082d45-3b3e-446f-829e-1dcb1eb71676\") " Oct 30 02:01:19.664676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.664647 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0082d45-3b3e-446f-829e-1dcb1eb71676-plugins-config-volume" pod "c0082d45-3b3e-446f-829e-1dcb1eb71676" (UID: "c0082d45-3b3e-446f-829e-1dcb1eb71676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:19.664960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.664939 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0082d45-3b3e-446f-829e-1dcb1eb71676-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:19.671122 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.666572 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0082d45-3b3e-446f-829e-1dcb1eb71676-kube-api-access-nrmf2" pod "c0082d45-3b3e-446f-829e-1dcb1eb71676" (UID: "c0082d45-3b3e-446f-829e-1dcb1eb71676"). InnerVolumeSpecName "kube-api-access-nrmf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:19.749756 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.749715 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:01:19.750056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.750029 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 02:01:19.750163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.750099 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 02:01:19.766316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.766232 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a06db84-72df-4a11-8164-0353070babca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a06db84-72df-4a11-8164-0353070babca-plugins-config-volume\") pod \"1a06db84-72df-4a11-8164-0353070babca\" (UID: \"1a06db84-72df-4a11-8164-0353070babca\") " Oct 30 02:01:19.766480 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.766316 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a06db84-72df-4a11-8164-0353070babca-kube-api-access-pnnkw\" (UniqueName: \"kubernetes.io/projected/1a06db84-72df-4a11-8164-0353070babca-kube-api-access-pnnkw\") pod \"1a06db84-72df-4a11-8164-0353070babca\" (UID: \"1a06db84-72df-4a11-8164-0353070babca\") " Oct 30 02:01:19.780110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.767966 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrmf2\" (UniqueName: \"kubernetes.io/projected/c0082d45-3b3e-446f-829e-1dcb1eb71676-kube-api-access-nrmf2\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:19.785581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.785410 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a06db84-72df-4a11-8164-0353070babca-plugins-config-volume" pod "1a06db84-72df-4a11-8164-0353070babca" (UID: "1a06db84-72df-4a11-8164-0353070babca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:19.800101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.800047 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a06db84-72df-4a11-8164-0353070babca-kube-api-access-pnnkw" pod "1a06db84-72df-4a11-8164-0353070babca" (UID: "1a06db84-72df-4a11-8164-0353070babca"). InnerVolumeSpecName "kube-api-access-pnnkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:19.893235 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.893176 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a06db84-72df-4a11-8164-0353070babca-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:19.893564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:19.893544 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnnkw\" (UniqueName: \"kubernetes.io/projected/1a06db84-72df-4a11-8164-0353070babca-kube-api-access-pnnkw\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.555464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.555370 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" Oct 30 02:01:20.555464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.555370 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb" Oct 30 02:01:20.555464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.555339 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w" event={"ID":"1a06db84-72df-4a11-8164-0353070babca","Type":"ContainerDied","Data":"6480dc2316714c652dec4e52cef2daeecec49599ee6bf649161711e0cafac732"} Oct 30 02:01:20.555464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.555458 5035 scope.go:122] "RemoveContainer" containerID="728dda8fbe7b8cdfa56c233379582ce533db2aa9cf57b01f29936782f3e5b1e6" Oct 30 02:01:20.656987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.656950 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb"] Oct 30 02:01:20.703637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.703598 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b273b6-6bcd8bb886-4mhbb"] Oct 30 02:01:20.724234 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.724197 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w"] Oct 30 02:01:20.729837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.729804 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b273b6-d5b5cd4bc-vp85w"] Oct 30 02:01:20.750852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.750811 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 02:01:20.751051 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:20.750878 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 02:01:21.662258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:21.662219 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a06db84-72df-4a11-8164-0353070babca" path="/var/lib/kubelet/pods/1a06db84-72df-4a11-8164-0353070babca/volumes" Oct 30 02:01:21.662934 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:21.662908 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0082d45-3b3e-446f-829e-1dcb1eb71676" path="/var/lib/kubelet/pods/c0082d45-3b3e-446f-829e-1dcb1eb71676/volumes" Oct 30 02:01:21.749884 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:21.749844 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 02:01:21.750080 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:21.749906 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 02:01:22.749897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:22.749856 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 02:01:22.750324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:22.749919 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 02:01:23.760554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:23.750199 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 02:01:23.760554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:23.750307 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 02:01:24.613342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:24.613291 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr"] Oct 30 02:01:24.750933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:24.750861 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 02:01:24.751147 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:24.750985 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 02:01:24.815465 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:24.815417 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:01:25.755681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:25.755646 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:01:25.756891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:25.756869 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:01:27.303631 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:27.303587 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925"] Oct 30 02:01:29.614675 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:29.614591 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" podUID="e3aeea64-7a64-4298-acff-44762f5835d4" containerName="payload-processing" containerID="cri-o://c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003" gracePeriod=25 Oct 30 02:01:29.947694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:29.947654 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:01:29.998668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:29.998276 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3aeea64-7a64-4298-acff-44762f5835d4-kube-api-access-lpx5k\" (UniqueName: \"kubernetes.io/projected/e3aeea64-7a64-4298-acff-44762f5835d4-kube-api-access-lpx5k\") pod \"e3aeea64-7a64-4298-acff-44762f5835d4\" (UID: \"e3aeea64-7a64-4298-acff-44762f5835d4\") " Oct 30 02:01:30.000256 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:29.999038 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3aeea64-7a64-4298-acff-44762f5835d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aeea64-7a64-4298-acff-44762f5835d4-plugins-config-volume\") pod \"e3aeea64-7a64-4298-acff-44762f5835d4\" (UID: \"e3aeea64-7a64-4298-acff-44762f5835d4\") " Oct 30 02:01:30.000863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.000602 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3aeea64-7a64-4298-acff-44762f5835d4-plugins-config-volume" pod "e3aeea64-7a64-4298-acff-44762f5835d4" (UID: "e3aeea64-7a64-4298-acff-44762f5835d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:30.002358 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.002314 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aeea64-7a64-4298-acff-44762f5835d4-kube-api-access-lpx5k" pod "e3aeea64-7a64-4298-acff-44762f5835d4" (UID: "e3aeea64-7a64-4298-acff-44762f5835d4"). InnerVolumeSpecName "kube-api-access-lpx5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:30.101107 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.101059 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpx5k\" (UniqueName: \"kubernetes.io/projected/e3aeea64-7a64-4298-acff-44762f5835d4-kube-api-access-lpx5k\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.101107 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.101097 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aeea64-7a64-4298-acff-44762f5835d4-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.707975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.707935 5035 generic.go:356] "Generic (PLEG): container finished" podID="e3aeea64-7a64-4298-acff-44762f5835d4" containerID="c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003" exitCode=0 Oct 30 02:01:30.708456 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.707998 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" Oct 30 02:01:30.708456 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.708018 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" event={"ID":"e3aeea64-7a64-4298-acff-44762f5835d4","Type":"ContainerDied","Data":"c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003"} Oct 30 02:01:30.708456 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.708077 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr" event={"ID":"e3aeea64-7a64-4298-acff-44762f5835d4","Type":"ContainerDied","Data":"3684b4c7560e951c4ed02c0174b9940dff44ef56e8eb792bfc7922ff2825a47c"} Oct 30 02:01:30.708456 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.708099 5035 scope.go:122] "RemoveContainer" containerID="c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003" Oct 30 02:01:30.723000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.722974 5035 scope.go:122] "RemoveContainer" containerID="c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003" Oct 30 02:01:30.723370 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:01:30.723348 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003\": container with ID starting with c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003 not found: ID does not exist" containerID="c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003" Oct 30 02:01:30.723429 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.723380 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003"} err="failed to get container status \"c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003\": rpc error: code = NotFound desc = could not find container \"c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003\": container with ID starting with c852b0cdeb63f798f96c07910d7ba4cb9fd572ea0fb048a3d1d3c7bb682a9003 not found: ID does not exist" Oct 30 02:01:30.765550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.765488 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr"] Oct 30 02:01:30.770265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:30.770227 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-145336-5f79ffd9c4-z2vbr"] Oct 30 02:01:31.646749 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:31.646714 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3aeea64-7a64-4298-acff-44762f5835d4" path="/var/lib/kubelet/pods/e3aeea64-7a64-4298-acff-44762f5835d4/volumes" Oct 30 02:01:32.304014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.303934 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" podUID="6e7654c0-f569-4dbd-a6e4-52cfa6cc8180" containerName="payload-pre-processing" containerID="cri-o://c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363" gracePeriod=25 Oct 30 02:01:32.613537 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.613497 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:01:32.735174 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.735137 5035 generic.go:356] "Generic (PLEG): container finished" podID="0a0da372-b63d-4050-899c-eb5007647157" containerID="145e6d1c1b67436098570cd37cee3ada549268458362fd9c43fe0ad96095a0e5" exitCode=0 Oct 30 02:01:32.735377 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.735209 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" event={"ID":"0a0da372-b63d-4050-899c-eb5007647157","Type":"ContainerDied","Data":"145e6d1c1b67436098570cd37cee3ada549268458362fd9c43fe0ad96095a0e5"} Oct 30 02:01:32.735819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.735795 5035 scope.go:122] "RemoveContainer" containerID="145e6d1c1b67436098570cd37cee3ada549268458362fd9c43fe0ad96095a0e5" Oct 30 02:01:32.737019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.736993 5035 generic.go:356] "Generic (PLEG): container finished" podID="6e7654c0-f569-4dbd-a6e4-52cfa6cc8180" containerID="c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363" exitCode=0 Oct 30 02:01:32.737112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.737031 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" Oct 30 02:01:32.737112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.737035 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" event={"ID":"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180","Type":"ContainerDied","Data":"c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363"} Oct 30 02:01:32.737112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.737084 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925" event={"ID":"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180","Type":"ContainerDied","Data":"159ce8661eb043ff501e62e07c3f464e7a8570467ae1cad00fcc0ada46af5ff5"} Oct 30 02:01:32.737268 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.737114 5035 scope.go:122] "RemoveContainer" containerID="c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363" Oct 30 02:01:32.737669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.737630 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:32.737777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.737727 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:32.737822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.737782 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:01:32.753591 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.753566 5035 scope.go:122] "RemoveContainer" containerID="c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363" Oct 30 02:01:32.753953 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:01:32.753924 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363\": container with ID starting with c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363 not found: ID does not exist" containerID="c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363" Oct 30 02:01:32.754024 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.753963 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363"} err="failed to get container status \"c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363\": rpc error: code = NotFound desc = could not find container \"c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363\": container with ID starting with c64478fa215c01d12812b6209ac55ae8f59987fa8ff6f16e86d6a7481c6d9363 not found: ID does not exist" Oct 30 02:01:32.768417 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.768392 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-plugins-config-volume\") pod \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\" (UID: \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\") " Oct 30 02:01:32.768566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.768515 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-kube-api-access-v28bh\" (UniqueName: \"kubernetes.io/projected/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-kube-api-access-v28bh\") pod \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\" (UID: \"6e7654c0-f569-4dbd-a6e4-52cfa6cc8180\") " Oct 30 02:01:32.768781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.768756 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-plugins-config-volume" pod "6e7654c0-f569-4dbd-a6e4-52cfa6cc8180" (UID: "6e7654c0-f569-4dbd-a6e4-52cfa6cc8180"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:32.769226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.769207 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.770649 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.770626 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-kube-api-access-v28bh" pod "6e7654c0-f569-4dbd-a6e4-52cfa6cc8180" (UID: "6e7654c0-f569-4dbd-a6e4-52cfa6cc8180"). InnerVolumeSpecName "kube-api-access-v28bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:32.826218 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.826187 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh"] Oct 30 02:01:32.863632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.862449 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx"] Oct 30 02:01:32.870725 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:32.870606 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v28bh\" (UniqueName: \"kubernetes.io/projected/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180-kube-api-access-v28bh\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:33.100219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:33.100177 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925"] Oct 30 02:01:33.104209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:33.104173 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38bc0d-5576b45767-4d925"] Oct 30 02:01:33.649924 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:33.649879 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7654c0-f569-4dbd-a6e4-52cfa6cc8180" path="/var/lib/kubelet/pods/6e7654c0-f569-4dbd-a6e4-52cfa6cc8180/volumes" Oct 30 02:01:33.751096 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:33.751039 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" event={"ID":"0a0da372-b63d-4050-899c-eb5007647157","Type":"ContainerStarted","Data":"8d79d33b355339578111d585e523fa4901f916f015a3a64b69b3cc3345626c12"} Oct 30 02:01:34.686555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:34.683000 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:01:34.693240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:34.693203 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:01:34.693580 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:34.693263 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 02:01:35.692462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:35.692419 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:01:35.693045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:35.692486 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 02:01:36.330665 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:36.330621 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:01:36.691826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:36.691717 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:01:36.691826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:36.691794 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 02:01:37.691602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:37.685967 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:01:37.691602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:37.686032 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 02:01:37.826691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:37.826568 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" podUID="c0540264-0f4b-4a01-82db-5eb14a142273" containerName="payload-processing" containerID="cri-o://bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff" gracePeriod=25 Oct 30 02:01:37.863279 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:37.863198 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" podUID="90c5206e-3ba7-4478-98a5-5dae3ff8f11d" containerName="payload-pre-processing" containerID="cri-o://db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8" gracePeriod=25 Oct 30 02:01:38.230940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.230020 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:01:38.304887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.304854 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:01:38.321088 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.321039 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0540264-0f4b-4a01-82db-5eb14a142273-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0540264-0f4b-4a01-82db-5eb14a142273-plugins-config-volume\") pod \"c0540264-0f4b-4a01-82db-5eb14a142273\" (UID: \"c0540264-0f4b-4a01-82db-5eb14a142273\") " Oct 30 02:01:38.321295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.321263 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0540264-0f4b-4a01-82db-5eb14a142273-kube-api-access-g9cnv\" (UniqueName: \"kubernetes.io/projected/c0540264-0f4b-4a01-82db-5eb14a142273-kube-api-access-g9cnv\") pod \"c0540264-0f4b-4a01-82db-5eb14a142273\" (UID: \"c0540264-0f4b-4a01-82db-5eb14a142273\") " Oct 30 02:01:38.321625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.321517 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0540264-0f4b-4a01-82db-5eb14a142273-plugins-config-volume" pod "c0540264-0f4b-4a01-82db-5eb14a142273" (UID: "c0540264-0f4b-4a01-82db-5eb14a142273"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:38.322022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.322000 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0540264-0f4b-4a01-82db-5eb14a142273-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:38.324338 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.324311 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0540264-0f4b-4a01-82db-5eb14a142273-kube-api-access-g9cnv" pod "c0540264-0f4b-4a01-82db-5eb14a142273" (UID: "c0540264-0f4b-4a01-82db-5eb14a142273"). InnerVolumeSpecName "kube-api-access-g9cnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:38.422846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.422789 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-plugins-config-volume\") pod \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\" (UID: \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\") " Oct 30 02:01:38.423066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.423036 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-kube-api-access-8z5sc\" (UniqueName: \"kubernetes.io/projected/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-kube-api-access-8z5sc\") pod \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\" (UID: \"90c5206e-3ba7-4478-98a5-5dae3ff8f11d\") " Oct 30 02:01:38.423275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.423246 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-plugins-config-volume" pod "90c5206e-3ba7-4478-98a5-5dae3ff8f11d" (UID: "90c5206e-3ba7-4478-98a5-5dae3ff8f11d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:38.423840 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.423815 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:38.423840 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.423846 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9cnv\" (UniqueName: \"kubernetes.io/projected/c0540264-0f4b-4a01-82db-5eb14a142273-kube-api-access-g9cnv\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:38.425592 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.425520 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-kube-api-access-8z5sc" pod "90c5206e-3ba7-4478-98a5-5dae3ff8f11d" (UID: "90c5206e-3ba7-4478-98a5-5dae3ff8f11d"). InnerVolumeSpecName "kube-api-access-8z5sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:38.525112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.524987 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z5sc\" (UniqueName: \"kubernetes.io/projected/90c5206e-3ba7-4478-98a5-5dae3ff8f11d-kube-api-access-8z5sc\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:38.688549 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.686098 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 02:01:38.688549 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.686186 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 02:01:38.823726 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.823630 5035 generic.go:356] "Generic (PLEG): container finished" podID="c0540264-0f4b-4a01-82db-5eb14a142273" containerID="bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff" exitCode=0 Oct 30 02:01:38.823726 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.823679 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" Oct 30 02:01:38.823726 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.823715 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" event={"ID":"c0540264-0f4b-4a01-82db-5eb14a142273","Type":"ContainerDied","Data":"bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff"} Oct 30 02:01:38.824287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.823771 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh" event={"ID":"c0540264-0f4b-4a01-82db-5eb14a142273","Type":"ContainerDied","Data":"f8a6301ff92f03201f0f7669594cbecee2cee957b004fa9bd62e8ec509e429d6"} Oct 30 02:01:38.824287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.823794 5035 scope.go:122] "RemoveContainer" containerID="bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff" Oct 30 02:01:38.825670 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.825641 5035 generic.go:356] "Generic (PLEG): container finished" podID="90c5206e-3ba7-4478-98a5-5dae3ff8f11d" containerID="db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8" exitCode=0 Oct 30 02:01:38.825796 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.825697 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" Oct 30 02:01:38.825796 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.825714 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" event={"ID":"90c5206e-3ba7-4478-98a5-5dae3ff8f11d","Type":"ContainerDied","Data":"db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8"} Oct 30 02:01:38.825796 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.825761 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx" event={"ID":"90c5206e-3ba7-4478-98a5-5dae3ff8f11d","Type":"ContainerDied","Data":"4ebb26c598c66fd9fb5b1ad0ac3205ce26c5694719f62007e86614a681852e52"} Oct 30 02:01:38.836763 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.836564 5035 scope.go:122] "RemoveContainer" containerID="bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff" Oct 30 02:01:38.836927 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:01:38.836905 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff\": container with ID starting with bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff not found: ID does not exist" containerID="bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff" Oct 30 02:01:38.836982 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.836947 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff"} err="failed to get container status \"bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff\": rpc error: code = NotFound desc = could not find container \"bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff\": container with ID starting with bd65b66b6fd1c8921189e87c589ed9becc699294ce90a2bda494375db29596ff not found: ID does not exist" Oct 30 02:01:38.836982 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.836967 5035 scope.go:122] "RemoveContainer" containerID="db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8" Oct 30 02:01:38.860024 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.859992 5035 scope.go:122] "RemoveContainer" containerID="db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8" Oct 30 02:01:38.860419 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:01:38.860396 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8\": container with ID starting with db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8 not found: ID does not exist" containerID="db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8" Oct 30 02:01:38.860508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.860425 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8"} err="failed to get container status \"db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8\": rpc error: code = NotFound desc = could not find container \"db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8\": container with ID starting with db9cf7db888c874bb9eb21d218c30dd8193c882e020b5c789dc882d61cb91bc8 not found: ID does not exist" Oct 30 02:01:38.906387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.906344 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx"] Oct 30 02:01:38.912171 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.912134 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-356082-6f945d9cbc-w4tbx"] Oct 30 02:01:38.953701 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.953658 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh"] Oct 30 02:01:38.957485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:38.957454 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-356082-84fd4794fb-578fh"] Oct 30 02:01:39.644620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:39.644583 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c5206e-3ba7-4478-98a5-5dae3ff8f11d" path="/var/lib/kubelet/pods/90c5206e-3ba7-4478-98a5-5dae3ff8f11d/volumes" Oct 30 02:01:39.645068 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:39.645054 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0540264-0f4b-4a01-82db-5eb14a142273" path="/var/lib/kubelet/pods/c0540264-0f4b-4a01-82db-5eb14a142273/volumes" Oct 30 02:01:39.688129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:39.688096 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:01:39.689455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:39.689424 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:01:47.631042 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:01:47.630999 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:01:56.243882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.243845 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5"] Oct 30 02:01:56.244423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.244127 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" podUID="90193f8e-9356-41c1-b6ba-8e465cdeead7" containerName="maas-api" containerID="cri-o://e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622" gracePeriod=30 Oct 30 02:01:56.638475 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.638443 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 02:01:56.648767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.648733 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90193f8e-9356-41c1-b6ba-8e465cdeead7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90193f8e-9356-41c1-b6ba-8e465cdeead7-openshift-service-ca\") pod \"90193f8e-9356-41c1-b6ba-8e465cdeead7\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " Oct 30 02:01:56.648941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.648786 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90193f8e-9356-41c1-b6ba-8e465cdeead7-kube-api-access-fhdsb\" (UniqueName: \"kubernetes.io/projected/90193f8e-9356-41c1-b6ba-8e465cdeead7-kube-api-access-fhdsb\") pod \"90193f8e-9356-41c1-b6ba-8e465cdeead7\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " Oct 30 02:01:56.648941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.648891 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls\") pod \"90193f8e-9356-41c1-b6ba-8e465cdeead7\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " Oct 30 02:01:56.649055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.648945 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-metrics-certs\") pod \"90193f8e-9356-41c1-b6ba-8e465cdeead7\" (UID: \"90193f8e-9356-41c1-b6ba-8e465cdeead7\") " Oct 30 02:01:56.649167 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.649145 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90193f8e-9356-41c1-b6ba-8e465cdeead7-openshift-service-ca" pod "90193f8e-9356-41c1-b6ba-8e465cdeead7" (UID: "90193f8e-9356-41c1-b6ba-8e465cdeead7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:56.649759 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.649731 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90193f8e-9356-41c1-b6ba-8e465cdeead7-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:56.651394 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.651366 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90193f8e-9356-41c1-b6ba-8e465cdeead7-kube-api-access-fhdsb" pod "90193f8e-9356-41c1-b6ba-8e465cdeead7" (UID: "90193f8e-9356-41c1-b6ba-8e465cdeead7"). InnerVolumeSpecName "kube-api-access-fhdsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:56.651683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.651657 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-metrics-certs" pod "90193f8e-9356-41c1-b6ba-8e465cdeead7" (UID: "90193f8e-9356-41c1-b6ba-8e465cdeead7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:56.651683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.651670 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls" pod "90193f8e-9356-41c1-b6ba-8e465cdeead7" (UID: "90193f8e-9356-41c1-b6ba-8e465cdeead7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:56.765522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.765478 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhdsb\" (UniqueName: \"kubernetes.io/projected/90193f8e-9356-41c1-b6ba-8e465cdeead7-kube-api-access-fhdsb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:56.765893 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.765856 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:56.766007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:56.765995 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90193f8e-9356-41c1-b6ba-8e465cdeead7-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:01:57.001468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.001428 5035 generic.go:356] "Generic (PLEG): container finished" podID="90193f8e-9356-41c1-b6ba-8e465cdeead7" containerID="e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622" exitCode=0 Oct 30 02:01:57.001687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.001480 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" Oct 30 02:01:57.001687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.001521 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" event={"ID":"90193f8e-9356-41c1-b6ba-8e465cdeead7","Type":"ContainerDied","Data":"e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622"} Oct 30 02:01:57.001687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.001600 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5" event={"ID":"90193f8e-9356-41c1-b6ba-8e465cdeead7","Type":"ContainerDied","Data":"fd67b3f3ab5729b2e1460781b5f041116c754a1d1aa24d20d5740e7ae1a6adad"} Oct 30 02:01:57.001687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.001632 5035 scope.go:122] "RemoveContainer" containerID="e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622" Oct 30 02:01:57.013916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.013887 5035 scope.go:122] "RemoveContainer" containerID="e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622" Oct 30 02:01:57.014283 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:01:57.014258 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622\": container with ID starting with e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622 not found: ID does not exist" containerID="e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622" Oct 30 02:01:57.014378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.014295 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622"} err="failed to get container status \"e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622\": rpc error: code = NotFound desc = could not find container \"e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622\": container with ID starting with e5f7cfb56082054d66ce26a4992724cd1d5cf2eb1f447a48fecabd5141c3f622 not found: ID does not exist" Oct 30 02:01:57.060542 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.060485 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5"] Oct 30 02:01:57.064263 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.064227 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6ba1d9-79dccf49cc-nhmw5"] Oct 30 02:01:57.644664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:57.644628 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90193f8e-9356-41c1-b6ba-8e465cdeead7" path="/var/lib/kubelet/pods/90193f8e-9356-41c1-b6ba-8e465cdeead7/volumes" Oct 30 02:01:59.523484 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:59.523446 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h"] Oct 30 02:01:59.524035 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:01:59.523718 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" containerID="cri-o://52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0" gracePeriod=30 Oct 30 02:02:01.797855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:01.797811 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr"] Oct 30 02:02:01.798542 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:01.798498 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" containerID="cri-o://06ba1b180bb62e556df1b79f5671fa0c0fb6da2efe35290482101925c19954be" gracePeriod=30 Oct 30 02:02:01.806749 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:01.806704 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn"] Oct 30 02:02:01.807315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:01.807234 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" containerID="cri-o://f91cf31fcd06e8fe92e66d3f7f4efc1173394ecf5666dcc3ab19942f290b34dc" gracePeriod=30 Oct 30 02:02:03.735015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.734972 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl"] Oct 30 02:02:03.736082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736057 5035 cpu_manager.go:414] "containerMap: removing container" podUID="90c5206e-3ba7-4478-98a5-5dae3ff8f11d" containerName="payload-pre-processing" Oct 30 02:02:03.736182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736084 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c5206e-3ba7-4478-98a5-5dae3ff8f11d" containerName="payload-pre-processing" Oct 30 02:02:03.736182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736118 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1a06db84-72df-4a11-8164-0353070babca" containerName="payload-pre-processing" Oct 30 02:02:03.736182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736126 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a06db84-72df-4a11-8164-0353070babca" containerName="payload-pre-processing" Oct 30 02:02:03.736182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736140 5035 cpu_manager.go:414] "containerMap: removing container" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" Oct 30 02:02:03.736182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736148 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" Oct 30 02:02:03.736182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736175 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerName="istio-proxy" Oct 30 02:02:03.736182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736183 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerName="istio-proxy" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736196 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c0540264-0f4b-4a01-82db-5eb14a142273" containerName="payload-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736203 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0540264-0f4b-4a01-82db-5eb14a142273" containerName="payload-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736220 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e3aeea64-7a64-4298-acff-44762f5835d4" containerName="payload-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736227 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3aeea64-7a64-4298-acff-44762f5835d4" containerName="payload-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736242 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c0082d45-3b3e-446f-829e-1dcb1eb71676" containerName="payload-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736250 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0082d45-3b3e-446f-829e-1dcb1eb71676" containerName="payload-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736266 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6e7654c0-f569-4dbd-a6e4-52cfa6cc8180" containerName="payload-pre-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736274 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7654c0-f569-4dbd-a6e4-52cfa6cc8180" containerName="payload-pre-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736312 5035 cpu_manager.go:414] "containerMap: removing container" podUID="90193f8e-9356-41c1-b6ba-8e465cdeead7" containerName="maas-api" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736321 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90193f8e-9356-41c1-b6ba-8e465cdeead7" containerName="maas-api" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736503 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3aeea64-7a64-4298-acff-44762f5835d4" containerName="payload-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736519 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0082d45-3b3e-446f-829e-1dcb1eb71676" containerName="payload-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736542 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7654c0-f569-4dbd-a6e4-52cfa6cc8180" containerName="payload-pre-processing" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736560 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="90193f8e-9356-41c1-b6ba-8e465cdeead7" containerName="maas-api" Oct 30 02:02:03.736570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736574 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c5206e-3ba7-4478-98a5-5dae3ff8f11d" containerName="payload-pre-processing" Oct 30 02:02:03.737340 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736587 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a06db84-72df-4a11-8164-0353070babca" containerName="payload-pre-processing" Oct 30 02:02:03.737340 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736596 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c118c25-2881-4ec9-a079-fdae4c67f8fd" containerName="istio-proxy" Oct 30 02:02:03.737340 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736608 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0b842e-01f5-4bb9-9dcb-efc9577a6e45" containerName="istio-proxy" Oct 30 02:02:03.737340 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.736616 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0540264-0f4b-4a01-82db-5eb14a142273" containerName="payload-processing" Oct 30 02:02:03.765266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.764682 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.786579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.783734 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl"] Oct 30 02:02:03.792630 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.792596 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-70a069-openshift-default-dockercfg-2bzd9\"" Oct 30 02:02:03.903552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.877428 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-certs\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.903552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.877474 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-data\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.903552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.877561 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmgm\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-kube-api-access-7zmgm\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.903552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.877609 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-podinfo\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.903552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.877649 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-credential-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.903552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.877671 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istiod-ca-cert\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.903552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.877718 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-envoy\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.903552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.877766 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-token\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.903552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.877814 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.981078 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-certs\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.981147 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-data\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.981253 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmgm\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-kube-api-access-7zmgm\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.981323 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-podinfo\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.981491 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-credential-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.981569 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istiod-ca-cert\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.981728 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-envoy\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.981788 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-token\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.981862 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.982255 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-certs\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.982395 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.982881 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-data\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.983105 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-credential-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.984555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.983323 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istiod-ca-cert\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.988509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.985669 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-podinfo\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:03.993927 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:03.993886 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-envoy\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:04.011453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:04.011379 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-token\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:04.012812 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:04.012758 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmgm\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-kube-api-access-7zmgm\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:04.092184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:04.092123 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:04.454552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:04.454501 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl"] Oct 30 02:02:04.456345 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:02:04.456303 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b81a0ab_3b26_403b_bfb3_b36cf060240f.slice/crio-7e990cc6c1f962e0532fdfee621c5d5946047b4581d1d28acfa48d58a2f6da50 WatchSource:0}: Error finding container 7e990cc6c1f962e0532fdfee621c5d5946047b4581d1d28acfa48d58a2f6da50: Status 404 returned error can't find the container with id 7e990cc6c1f962e0532fdfee621c5d5946047b4581d1d28acfa48d58a2f6da50 Oct 30 02:02:04.459336 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:04.459294 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:02:04.459463 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:04.459370 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:02:04.459463 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:04.459401 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:02:04.883441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:04.883409 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.010442 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-istio-token\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-istio-token\") pod \"3879878f-0211-4689-b111-62aead00bccf\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.010520 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-socket\") pod \"3879878f-0211-4689-b111-62aead00bccf\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.011204 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-envoy\") pod \"3879878f-0211-4689-b111-62aead00bccf\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.011300 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3879878f-0211-4689-b111-62aead00bccf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879878f-0211-4689-b111-62aead00bccf-istiod-ca-cert\") pod \"3879878f-0211-4689-b111-62aead00bccf\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.011372 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-credential-socket\") pod \"3879878f-0211-4689-b111-62aead00bccf\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.011419 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-kube-api-access-drgkr\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-kube-api-access-drgkr\") pod \"3879878f-0211-4689-b111-62aead00bccf\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.011476 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3879878f-0211-4689-b111-62aead00bccf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879878f-0211-4689-b111-62aead00bccf-istio-podinfo\") pod \"3879878f-0211-4689-b111-62aead00bccf\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.011503 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-certs\") pod \"3879878f-0211-4689-b111-62aead00bccf\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.011630 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-data\") pod \"3879878f-0211-4689-b111-62aead00bccf\" (UID: \"3879878f-0211-4689-b111-62aead00bccf\") " Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.013423 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-socket" pod "3879878f-0211-4689-b111-62aead00bccf" (UID: "3879878f-0211-4689-b111-62aead00bccf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.013887 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3879878f-0211-4689-b111-62aead00bccf-istiod-ca-cert" pod "3879878f-0211-4689-b111-62aead00bccf" (UID: "3879878f-0211-4689-b111-62aead00bccf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.014241 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-credential-socket" pod "3879878f-0211-4689-b111-62aead00bccf" (UID: "3879878f-0211-4689-b111-62aead00bccf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.014324 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-certs" pod "3879878f-0211-4689-b111-62aead00bccf" (UID: "3879878f-0211-4689-b111-62aead00bccf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.014594 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-data" pod "3879878f-0211-4689-b111-62aead00bccf" (UID: "3879878f-0211-4689-b111-62aead00bccf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.024269 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.024304 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.024318 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.024333 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879878f-0211-4689-b111-62aead00bccf-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.024347 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.064605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.062810 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-envoy" pod "3879878f-0211-4689-b111-62aead00bccf" (UID: "3879878f-0211-4689-b111-62aead00bccf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:05.073623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.073584 5035 generic.go:356] "Generic (PLEG): container finished" podID="3879878f-0211-4689-b111-62aead00bccf" containerID="52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0" exitCode=0 Oct 30 02:02:05.073798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.073779 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" Oct 30 02:02:05.074292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.074262 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" event={"ID":"3879878f-0211-4689-b111-62aead00bccf","Type":"ContainerDied","Data":"52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0"} Oct 30 02:02:05.074374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.074328 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h" event={"ID":"3879878f-0211-4689-b111-62aead00bccf","Type":"ContainerDied","Data":"e1a0d4a4eb94645d5c918b88026d0de3eecbd7a51627ef9248f876868e305f17"} Oct 30 02:02:05.074374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.074363 5035 scope.go:122] "RemoveContainer" containerID="52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0" Oct 30 02:02:05.081484 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.081438 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" event={"ID":"1b81a0ab-3b26-403b-bfb3-b36cf060240f","Type":"ContainerStarted","Data":"965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67"} Oct 30 02:02:05.081655 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.081556 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" event={"ID":"1b81a0ab-3b26-403b-bfb3-b36cf060240f","Type":"ContainerStarted","Data":"7e990cc6c1f962e0532fdfee621c5d5946047b4581d1d28acfa48d58a2f6da50"} Oct 30 02:02:05.085294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.085242 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-kube-api-access-drgkr" pod "3879878f-0211-4689-b111-62aead00bccf" (UID: "3879878f-0211-4689-b111-62aead00bccf"). InnerVolumeSpecName "kube-api-access-drgkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:05.086877 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.086846 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3879878f-0211-4689-b111-62aead00bccf-istio-podinfo" pod "3879878f-0211-4689-b111-62aead00bccf" (UID: "3879878f-0211-4689-b111-62aead00bccf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:05.087030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.086992 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-istio-token" pod "3879878f-0211-4689-b111-62aead00bccf" (UID: "3879878f-0211-4689-b111-62aead00bccf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:05.100495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.100130 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:05.107544 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.105900 5035 scope.go:122] "RemoveContainer" containerID="cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684" Oct 30 02:02:05.108336 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.108282 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:05.126405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.126367 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.126405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.126402 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879878f-0211-4689-b111-62aead00bccf-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.126637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.126415 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drgkr\" (UniqueName: \"kubernetes.io/projected/3879878f-0211-4689-b111-62aead00bccf-kube-api-access-drgkr\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.126637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.126428 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879878f-0211-4689-b111-62aead00bccf-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.128742 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.128715 5035 scope.go:122] "RemoveContainer" containerID="52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0" Oct 30 02:02:05.129135 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:02:05.129111 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0\": container with ID starting with 52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0 not found: ID does not exist" containerID="52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0" Oct 30 02:02:05.129293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.129148 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0"} err="failed to get container status \"52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0\": rpc error: code = NotFound desc = could not find container \"52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0\": container with ID starting with 52659a6100912828ff2354874368056b46b371dcf5af62508208eb00f828b5d0 not found: ID does not exist" Oct 30 02:02:05.129293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.129177 5035 scope.go:122] "RemoveContainer" containerID="cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684" Oct 30 02:02:05.129512 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:02:05.129474 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684\": container with ID starting with cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684 not found: ID does not exist" containerID="cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684" Oct 30 02:02:05.129635 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.129505 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684"} err="failed to get container status \"cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684\": rpc error: code = NotFound desc = could not find container \"cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684\": container with ID starting with cab858dcb49dd2cd75be32828e2541f36a57e5bfeeaa32d84dc77a2c1a4cf684 not found: ID does not exist" Oct 30 02:02:05.141019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.140967 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" podStartSLOduration=2.140952031 podStartE2EDuration="2.140952031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:05.137710702 +0000 UTC m=+2003.640642921" watchObservedRunningTime="2026-10-30 02:02:05.140952031 +0000 UTC m=+2003.643884323" Oct 30 02:02:05.459979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.459946 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h"] Oct 30 02:02:05.470553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.470499 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-356082-openshift-default-7dd6879f-cf82h"] Oct 30 02:02:05.645347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:05.645306 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3879878f-0211-4689-b111-62aead00bccf" path="/var/lib/kubelet/pods/3879878f-0211-4689-b111-62aead00bccf/volumes" Oct 30 02:02:06.092135 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:06.092100 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:06.093299 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:06.093272 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:02:07.125837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.125796 5035 generic.go:356] "Generic (PLEG): container finished" podID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerID="b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9" exitCode=0 Oct 30 02:02:07.126359 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.125858 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" event={"ID":"2ae724a0-57d2-4116-9537-31e7aa2aea87","Type":"ContainerDied","Data":"b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9"} Oct 30 02:02:07.126448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.126389 5035 scope.go:122] "RemoveContainer" containerID="b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9" Oct 30 02:02:07.128389 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.128357 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:02:07.128512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.128453 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:02:07.128512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.128499 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:02:07.128645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.128579 5035 generic.go:356] "Generic (PLEG): container finished" podID="27942cd1-bb9c-4470-89d2-89d82022090a" containerID="06ba1b180bb62e556df1b79f5671fa0c0fb6da2efe35290482101925c19954be" exitCode=0 Oct 30 02:02:07.128755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.128735 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" event={"ID":"27942cd1-bb9c-4470-89d2-89d82022090a","Type":"ContainerDied","Data":"06ba1b180bb62e556df1b79f5671fa0c0fb6da2efe35290482101925c19954be"} Oct 30 02:02:07.131141 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.131113 5035 generic.go:356] "Generic (PLEG): container finished" podID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerID="f91cf31fcd06e8fe92e66d3f7f4efc1173394ecf5666dcc3ab19942f290b34dc" exitCode=0 Oct 30 02:02:07.131268 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.131183 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" event={"ID":"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6","Type":"ContainerDied","Data":"f91cf31fcd06e8fe92e66d3f7f4efc1173394ecf5666dcc3ab19942f290b34dc"} Oct 30 02:02:07.134230 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.134206 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:02:07.180118 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.180085 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:02:07.262593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.262554 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-kube-api-access-6hzn7\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-kube-api-access-6hzn7\") pod \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " Oct 30 02:02:07.262951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.262926 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-data\") pod \"27942cd1-bb9c-4470-89d2-89d82022090a\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " Oct 30 02:02:07.263127 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.263110 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-envoy\") pod \"27942cd1-bb9c-4470-89d2-89d82022090a\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " Oct 30 02:02:07.263296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.263279 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-istio-token\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-istio-token\") pod \"27942cd1-bb9c-4470-89d2-89d82022090a\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " Oct 30 02:02:07.263461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.263446 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-socket\") pod \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " Oct 30 02:02:07.263609 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.263592 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-podinfo\") pod \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " Oct 30 02:02:07.263749 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.263731 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-certs\") pod \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " Oct 30 02:02:07.263943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.263923 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istiod-ca-cert\") pod \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " Oct 30 02:02:07.264086 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.264071 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27942cd1-bb9c-4470-89d2-89d82022090a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27942cd1-bb9c-4470-89d2-89d82022090a-istio-podinfo\") pod \"27942cd1-bb9c-4470-89d2-89d82022090a\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " Oct 30 02:02:07.264217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.264202 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-credential-socket\") pod \"27942cd1-bb9c-4470-89d2-89d82022090a\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " Oct 30 02:02:07.264346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.264330 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-credential-socket\") pod \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " Oct 30 02:02:07.264573 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.264552 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-socket\") pod \"27942cd1-bb9c-4470-89d2-89d82022090a\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " Oct 30 02:02:07.264761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.264743 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-kube-api-access-tbfv2\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-kube-api-access-tbfv2\") pod \"27942cd1-bb9c-4470-89d2-89d82022090a\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " Oct 30 02:02:07.264896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.264881 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-certs\") pod \"27942cd1-bb9c-4470-89d2-89d82022090a\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " Oct 30 02:02:07.265023 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.265008 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-envoy\") pod \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " Oct 30 02:02:07.265228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.265211 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-data\") pod \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " Oct 30 02:02:07.265380 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.265362 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-token\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-token\") pod \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\" (UID: \"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6\") " Oct 30 02:02:07.265798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.265485 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27942cd1-bb9c-4470-89d2-89d82022090a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27942cd1-bb9c-4470-89d2-89d82022090a-istiod-ca-cert\") pod \"27942cd1-bb9c-4470-89d2-89d82022090a\" (UID: \"27942cd1-bb9c-4470-89d2-89d82022090a\") " Oct 30 02:02:07.265903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.263187 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-data" pod "27942cd1-bb9c-4470-89d2-89d82022090a" (UID: "27942cd1-bb9c-4470-89d2-89d82022090a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.265972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.264544 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istiod-ca-cert" pod "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" (UID: "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:07.265972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.264762 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-socket" pod "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" (UID: "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.265972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.264949 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-credential-socket" pod "27942cd1-bb9c-4470-89d2-89d82022090a" (UID: "27942cd1-bb9c-4470-89d2-89d82022090a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.265972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.265115 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-certs" pod "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" (UID: "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.265972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.265250 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-socket" pod "27942cd1-bb9c-4470-89d2-89d82022090a" (UID: "27942cd1-bb9c-4470-89d2-89d82022090a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.265972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.265457 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-credential-socket" pod "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" (UID: "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.265972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.265632 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-certs" pod "27942cd1-bb9c-4470-89d2-89d82022090a" (UID: "27942cd1-bb9c-4470-89d2-89d82022090a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.266346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.266233 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-data" pod "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" (UID: "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.267179 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.267153 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.267316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.267302 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.267418 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.267405 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.267511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.267500 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.267635 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.267617 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.267727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.267714 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.267827 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.267815 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.268664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.267927 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.268664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.267946 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.268664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.268339 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-kube-api-access-tbfv2" pod "27942cd1-bb9c-4470-89d2-89d82022090a" (UID: "27942cd1-bb9c-4470-89d2-89d82022090a"). InnerVolumeSpecName "kube-api-access-tbfv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.271077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.269702 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27942cd1-bb9c-4470-89d2-89d82022090a-istio-podinfo" pod "27942cd1-bb9c-4470-89d2-89d82022090a" (UID: "27942cd1-bb9c-4470-89d2-89d82022090a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:07.271077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.270100 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-podinfo" pod "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" (UID: "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:07.271077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.270354 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-envoy" pod "27942cd1-bb9c-4470-89d2-89d82022090a" (UID: "27942cd1-bb9c-4470-89d2-89d82022090a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.271077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.270929 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-kube-api-access-6hzn7" pod "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" (UID: "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6"). InnerVolumeSpecName "kube-api-access-6hzn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.271077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.271045 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-envoy" pod "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" (UID: "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.271390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.271368 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-istio-token" pod "27942cd1-bb9c-4470-89d2-89d82022090a" (UID: "27942cd1-bb9c-4470-89d2-89d82022090a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.271644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.271620 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-token" pod "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" (UID: "cb2c413f-3ed0-4f81-9c68-ba40983d4ba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.271644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.271633 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27942cd1-bb9c-4470-89d2-89d82022090a-istiod-ca-cert" pod "27942cd1-bb9c-4470-89d2-89d82022090a" (UID: "27942cd1-bb9c-4470-89d2-89d82022090a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:07.369305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.369211 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfv2\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-kube-api-access-tbfv2\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.369305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.369247 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.369305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.369257 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.369305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.369267 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27942cd1-bb9c-4470-89d2-89d82022090a-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.369305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.369276 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzn7\" (UniqueName: \"kubernetes.io/projected/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-kube-api-access-6hzn7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.369305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.369285 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27942cd1-bb9c-4470-89d2-89d82022090a-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.369305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.369293 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27942cd1-bb9c-4470-89d2-89d82022090a-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.369305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.369303 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.369305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.369310 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27942cd1-bb9c-4470-89d2-89d82022090a-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.839565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.839508 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl"] Oct 30 02:02:07.840178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.840161 5035 cpu_manager.go:414] "containerMap: removing container" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" Oct 30 02:02:07.840253 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.840180 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" Oct 30 02:02:07.840253 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.840207 5035 cpu_manager.go:414] "containerMap: removing container" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" Oct 30 02:02:07.840253 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.840213 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" Oct 30 02:02:07.840253 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.840231 5035 cpu_manager.go:414] "containerMap: removing container" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" Oct 30 02:02:07.840253 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.840238 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" Oct 30 02:02:07.840411 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.840351 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" containerName="istio-proxy" Oct 30 02:02:07.840411 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.840361 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" Oct 30 02:02:07.840411 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.840369 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" containerName="istio-proxy" Oct 30 02:02:07.844132 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.844101 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:07.865908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.865872 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-168989-dockercfg-9qkzb\"" Oct 30 02:02:07.866576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.866495 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-168989\"" Oct 30 02:02:07.867365 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.867339 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl"] Oct 30 02:02:07.941161 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.938784 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n"] Oct 30 02:02:07.942573 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.942519 5035 cpu_manager.go:414] "containerMap: removing container" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" Oct 30 02:02:07.942770 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.942746 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" Oct 30 02:02:07.942976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.942962 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="3879878f-0211-4689-b111-62aead00bccf" containerName="istio-proxy" Oct 30 02:02:07.947447 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.947411 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:07.968949 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:07.968911 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n"] Oct 30 02:02:08.022506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.022420 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74405b1-8c63-4204-8c23-4d8540e7bdc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n\" (UID: \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:08.022962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.022927 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vwb\" (UniqueName: \"kubernetes.io/projected/b68152d0-8586-49d1-be82-1c5f8887936a-kube-api-access-84vwb\") pod \"payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl\" (UID: \"b68152d0-8586-49d1-be82-1c5f8887936a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:08.023218 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.023197 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqpxv\" (UniqueName: \"kubernetes.io/projected/c74405b1-8c63-4204-8c23-4d8540e7bdc4-kube-api-access-qqpxv\") pod \"payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n\" (UID: \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:08.076060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.076005 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68152d0-8586-49d1-be82-1c5f8887936a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl\" (UID: \"b68152d0-8586-49d1-be82-1c5f8887936a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:08.140690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.140650 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" event={"ID":"2ae724a0-57d2-4116-9537-31e7aa2aea87","Type":"ContainerStarted","Data":"f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62"} Oct 30 02:02:08.142477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.142455 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" Oct 30 02:02:08.142614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.142488 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr" event={"ID":"27942cd1-bb9c-4470-89d2-89d82022090a","Type":"ContainerDied","Data":"8f656e0d5e389dba0323fadb9b3a6427335f7278e90acc3498883341f8c64b83"} Oct 30 02:02:08.142614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.142596 5035 scope.go:122] "RemoveContainer" containerID="06ba1b180bb62e556df1b79f5671fa0c0fb6da2efe35290482101925c19954be" Oct 30 02:02:08.144258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.144237 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" Oct 30 02:02:08.144389 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.144236 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn" event={"ID":"cb2c413f-3ed0-4f81-9c68-ba40983d4ba6","Type":"ContainerDied","Data":"00ab2fc7a0ba3d9c7a7333ef4c1c7b9ae0231e333500adc449d32184d6649fa7"} Oct 30 02:02:08.154126 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.154102 5035 scope.go:122] "RemoveContainer" containerID="f91cf31fcd06e8fe92e66d3f7f4efc1173394ecf5666dcc3ab19942f290b34dc" Oct 30 02:02:08.177592 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.177501 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68152d0-8586-49d1-be82-1c5f8887936a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl\" (UID: \"b68152d0-8586-49d1-be82-1c5f8887936a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:08.178640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.177783 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74405b1-8c63-4204-8c23-4d8540e7bdc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n\" (UID: \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:08.178640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.177907 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vwb\" (UniqueName: \"kubernetes.io/projected/b68152d0-8586-49d1-be82-1c5f8887936a-kube-api-access-84vwb\") pod \"payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl\" (UID: \"b68152d0-8586-49d1-be82-1c5f8887936a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:08.178640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.178024 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqpxv\" (UniqueName: \"kubernetes.io/projected/c74405b1-8c63-4204-8c23-4d8540e7bdc4-kube-api-access-qqpxv\") pod \"payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n\" (UID: \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:08.178640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.178402 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68152d0-8586-49d1-be82-1c5f8887936a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl\" (UID: \"b68152d0-8586-49d1-be82-1c5f8887936a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:08.179050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.179032 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74405b1-8c63-4204-8c23-4d8540e7bdc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n\" (UID: \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:08.212762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.212724 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqpxv\" (UniqueName: \"kubernetes.io/projected/c74405b1-8c63-4204-8c23-4d8540e7bdc4-kube-api-access-qqpxv\") pod \"payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n\" (UID: \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:08.215569 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.215523 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vwb\" (UniqueName: \"kubernetes.io/projected/b68152d0-8586-49d1-be82-1c5f8887936a-kube-api-access-84vwb\") pod \"payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl\" (UID: \"b68152d0-8586-49d1-be82-1c5f8887936a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:08.241654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.241613 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr"] Oct 30 02:02:08.251638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.251597 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-qzkrr"] Oct 30 02:02:08.296065 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.296031 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn"] Oct 30 02:02:08.301781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.301742 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-zg4xn"] Oct 30 02:02:08.322436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.322400 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f"] Oct 30 02:02:08.330317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.330287 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.348157 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.348127 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ea4bc0-openshift-default-dockercfg-fnj7b\"" Oct 30 02:02:08.351281 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.351252 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f"] Oct 30 02:02:08.390872 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.390829 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:08.487651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.487280 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:08.502804 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.502758 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-podinfo\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.503182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.503138 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-socket\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.503432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.503415 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-data\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.503776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.503756 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-token\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.504003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.503986 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-credential-socket\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.504198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.504180 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-envoy\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.504420 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.504404 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46kkc\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-kube-api-access-46kkc\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.511797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.511695 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istiod-ca-cert\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.512111 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.512086 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-certs\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.629880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.629158 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-podinfo\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.629880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.629815 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-socket\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.630134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.630040 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-data\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.630134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.630121 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-token\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.630252 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.630192 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-credential-socket\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.630311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.630264 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-envoy\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.630480 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.630456 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46kkc\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-kube-api-access-46kkc\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.630643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.630519 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istiod-ca-cert\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.630713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.630624 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-certs\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.631352 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.631258 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-certs\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.631571 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.631496 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-data\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.632219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.632169 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-credential-socket\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.632538 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.632474 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istiod-ca-cert\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.632815 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.632782 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-socket\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.635509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.635478 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-podinfo\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.636054 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.636001 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-envoy\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.720680 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.703303 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-token\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.730502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.725694 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46kkc\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-kube-api-access-46kkc\") pod \"e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.730502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.728761 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:02:08.730502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.728793 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 02:02:08.730502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.728850 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 02:02:08.947934 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.947877 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n"] Oct 30 02:02:08.949365 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:08.949332 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:08.950067 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:02:08.950038 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74405b1_8c63_4204_8c23_4d8540e7bdc4.slice/crio-c6a13b580e7e1b4c66a62b40c7345c4b42307ae39b3edab26b7f3ce0ab7e2106 WatchSource:0}: Error finding container c6a13b580e7e1b4c66a62b40c7345c4b42307ae39b3edab26b7f3ce0ab7e2106: Status 404 returned error can't find the container with id c6a13b580e7e1b4c66a62b40c7345c4b42307ae39b3edab26b7f3ce0ab7e2106 Oct 30 02:02:09.019874 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.019833 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl"] Oct 30 02:02:09.156916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.156873 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" event={"ID":"b68152d0-8586-49d1-be82-1c5f8887936a","Type":"ContainerStarted","Data":"0e2060847719c98c3a9e6bfc1c42bb608463ff9c79faa591b012502a6710db75"} Oct 30 02:02:09.161555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.161502 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" event={"ID":"c74405b1-8c63-4204-8c23-4d8540e7bdc4","Type":"ContainerStarted","Data":"b8a870c442f1c3e909a5ce185c79efb713b07014334e73be1db3ca8074c4c025"} Oct 30 02:02:09.161737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.161572 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" event={"ID":"c74405b1-8c63-4204-8c23-4d8540e7bdc4","Type":"ContainerStarted","Data":"c6a13b580e7e1b4c66a62b40c7345c4b42307ae39b3edab26b7f3ce0ab7e2106"} Oct 30 02:02:09.213327 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.213278 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" podStartSLOduration=2.213261378 podStartE2EDuration="2.213261378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:09.208680687 +0000 UTC m=+2007.711612907" watchObservedRunningTime="2026-10-30 02:02:09.213261378 +0000 UTC m=+2007.716193608" Oct 30 02:02:09.344336 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.344172 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f"] Oct 30 02:02:09.347164 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:02:09.347124 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fb784e_0d01_4def_bdc1_63259c9e1fbf.slice/crio-6445359c566fafc210cc379d669bab95c7903056feefca4025ba4701af4cc145 WatchSource:0}: Error finding container 6445359c566fafc210cc379d669bab95c7903056feefca4025ba4701af4cc145: Status 404 returned error can't find the container with id 6445359c566fafc210cc379d669bab95c7903056feefca4025ba4701af4cc145 Oct 30 02:02:09.350831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.350765 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:02:09.350965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.350848 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:02:09.350965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.350877 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:02:09.646709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.646625 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27942cd1-bb9c-4470-89d2-89d82022090a" path="/var/lib/kubelet/pods/27942cd1-bb9c-4470-89d2-89d82022090a/volumes" Oct 30 02:02:09.647126 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.647112 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2c413f-3ed0-4f81-9c68-ba40983d4ba6" path="/var/lib/kubelet/pods/cb2c413f-3ed0-4f81-9c68-ba40983d4ba6/volumes" Oct 30 02:02:09.729490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.729448 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 02:02:09.729716 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.729518 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 02:02:09.899963 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.899856 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k"] Oct 30 02:02:09.904406 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.904370 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:09.920996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.920966 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-70a069\"" Oct 30 02:02:09.923837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.923803 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k"] Oct 30 02:02:09.948125 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.948087 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c22c39-4909-447a-a55b-c20294cdcd1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k\" (UID: \"47c22c39-4909-447a-a55b-c20294cdcd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:09.948327 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:09.948196 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnk5\" (UniqueName: \"kubernetes.io/projected/47c22c39-4909-447a-a55b-c20294cdcd1f-kube-api-access-swnk5\") pod \"payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k\" (UID: \"47c22c39-4909-447a-a55b-c20294cdcd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:10.133586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.132323 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c22c39-4909-447a-a55b-c20294cdcd1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k\" (UID: \"47c22c39-4909-447a-a55b-c20294cdcd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:10.133586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.132493 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnk5\" (UniqueName: \"kubernetes.io/projected/47c22c39-4909-447a-a55b-c20294cdcd1f-kube-api-access-swnk5\") pod \"payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k\" (UID: \"47c22c39-4909-447a-a55b-c20294cdcd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:10.134117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.134080 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c22c39-4909-447a-a55b-c20294cdcd1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k\" (UID: \"47c22c39-4909-447a-a55b-c20294cdcd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:10.228598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.217787 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnk5\" (UniqueName: \"kubernetes.io/projected/47c22c39-4909-447a-a55b-c20294cdcd1f-kube-api-access-swnk5\") pod \"payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k\" (UID: \"47c22c39-4909-447a-a55b-c20294cdcd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:10.231933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.231877 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" event={"ID":"b68152d0-8586-49d1-be82-1c5f8887936a","Type":"ContainerStarted","Data":"d9f03ec2970db4a803af19ea4e5c1fd2c6abe6122fdab249eea7237a42b95d92"} Oct 30 02:02:10.232133 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.232062 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:10.234302 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.234275 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" event={"ID":"f5fb784e-0d01-4def-bdc1-63259c9e1fbf","Type":"ContainerStarted","Data":"348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84"} Oct 30 02:02:10.234458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.234316 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" event={"ID":"f5fb784e-0d01-4def-bdc1-63259c9e1fbf","Type":"ContainerStarted","Data":"6445359c566fafc210cc379d669bab95c7903056feefca4025ba4701af4cc145"} Oct 30 02:02:10.234458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.234445 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:10.332223 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.332166 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" podStartSLOduration=3.332148576 podStartE2EDuration="3.332148576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:10.282918016 +0000 UTC m=+2008.785850240" watchObservedRunningTime="2026-10-30 02:02:10.332148576 +0000 UTC m=+2008.835080903" Oct 30 02:02:10.333658 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.333604 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" podStartSLOduration=2.33358735 podStartE2EDuration="2.33358735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:10.33011904 +0000 UTC m=+2008.833051257" watchObservedRunningTime="2026-10-30 02:02:10.33358735 +0000 UTC m=+2008.836519571" Oct 30 02:02:10.524982 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.524155 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:10.729699 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.729666 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 02:02:10.729884 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.729719 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 02:02:10.950032 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.949998 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:10.957308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:10.957273 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:11.008281 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:11.008238 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k"] Oct 30 02:02:11.009819 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:02:11.009792 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c22c39_4909_447a_a55b_c20294cdcd1f.slice/crio-705c806c7a96bf59c8c45abec1c04eb1fdfe6717d9e4129e986f977e0e963754 WatchSource:0}: Error finding container 705c806c7a96bf59c8c45abec1c04eb1fdfe6717d9e4129e986f977e0e963754: Status 404 returned error can't find the container with id 705c806c7a96bf59c8c45abec1c04eb1fdfe6717d9e4129e986f977e0e963754 Oct 30 02:02:11.243385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:11.243281 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" event={"ID":"47c22c39-4909-447a-a55b-c20294cdcd1f","Type":"ContainerStarted","Data":"d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136"} Oct 30 02:02:11.243385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:11.243338 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" event={"ID":"47c22c39-4909-447a-a55b-c20294cdcd1f","Type":"ContainerStarted","Data":"705c806c7a96bf59c8c45abec1c04eb1fdfe6717d9e4129e986f977e0e963754"} Oct 30 02:02:11.243936 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:11.243769 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:11.244731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:11.244704 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:02:11.294342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:11.294272 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" podStartSLOduration=2.294249658 podStartE2EDuration="2.294249658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:11.287974346 +0000 UTC m=+2009.790906566" watchObservedRunningTime="2026-10-30 02:02:11.294249658 +0000 UTC m=+2009.797181881" Oct 30 02:02:11.728960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:11.728924 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 02:02:11.729164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:11.728997 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 02:02:12.252895 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:12.252858 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:12.728932 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:12.728890 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 02:02:12.729132 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:12.728949 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 02:02:13.728918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:13.728880 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 02:02:13.729481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:13.728945 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 02:02:14.733631 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.733592 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:02:14.734164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.734077 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:02:14.734786 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.734761 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:02:14.744097 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.744064 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb"] Oct 30 02:02:14.747638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.747619 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:14.768317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.768278 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ea4bc0\"" Oct 30 02:02:14.768513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.768405 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ea4bc0-dockercfg-jf7l2\"" Oct 30 02:02:14.774179 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.774135 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb"] Oct 30 02:02:14.916661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.900586 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6gq\" (UniqueName: \"kubernetes.io/projected/0ee74cc4-a5ee-4104-988a-9301ab5d7548-kube-api-access-7t6gq\") pod \"payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb\" (UID: \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:14.916661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:14.900655 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee74cc4-a5ee-4104-988a-9301ab5d7548-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb\" (UID: \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:15.002243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:15.002140 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6gq\" (UniqueName: \"kubernetes.io/projected/0ee74cc4-a5ee-4104-988a-9301ab5d7548-kube-api-access-7t6gq\") pod \"payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb\" (UID: \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:15.002243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:15.002208 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee74cc4-a5ee-4104-988a-9301ab5d7548-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb\" (UID: \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:15.005543 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:15.005475 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee74cc4-a5ee-4104-988a-9301ab5d7548-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb\" (UID: \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:15.031417 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:15.031380 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6gq\" (UniqueName: \"kubernetes.io/projected/0ee74cc4-a5ee-4104-988a-9301ab5d7548-kube-api-access-7t6gq\") pod \"payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb\" (UID: \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:15.063279 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:15.063244 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:15.381919 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:15.381884 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb"] Oct 30 02:02:15.382810 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:02:15.382781 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee74cc4_a5ee_4104_988a_9301ab5d7548.slice/crio-52b0e529bb5b4242f66243750c8072b7e691d16eaf422dc87a8a44f7bd3e1a12 WatchSource:0}: Error finding container 52b0e529bb5b4242f66243750c8072b7e691d16eaf422dc87a8a44f7bd3e1a12: Status 404 returned error can't find the container with id 52b0e529bb5b4242f66243750c8072b7e691d16eaf422dc87a8a44f7bd3e1a12 Oct 30 02:02:16.285061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.285024 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" event={"ID":"0ee74cc4-a5ee-4104-988a-9301ab5d7548","Type":"ContainerStarted","Data":"8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03"} Oct 30 02:02:16.285522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.285086 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" event={"ID":"0ee74cc4-a5ee-4104-988a-9301ab5d7548","Type":"ContainerStarted","Data":"52b0e529bb5b4242f66243750c8072b7e691d16eaf422dc87a8a44f7bd3e1a12"} Oct 30 02:02:16.285522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.285114 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:16.332702 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.332641 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" podStartSLOduration=2.3326256819999998 podStartE2EDuration="2.332625682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:16.328102143 +0000 UTC m=+2014.831034363" watchObservedRunningTime="2026-10-30 02:02:16.332625682 +0000 UTC m=+2014.835557902" Oct 30 02:02:16.533904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.533865 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn"] Oct 30 02:02:16.534099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.534078 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" podUID="2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" containerName="maas-api" containerID="cri-o://a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76" gracePeriod=30 Oct 30 02:02:16.866916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.866885 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 02:02:16.932172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.932136 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-maas-api-tls\") pod \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " Oct 30 02:02:16.932395 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.932365 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-kube-api-access-5cdrz\" (UniqueName: \"kubernetes.io/projected/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-kube-api-access-5cdrz\") pod \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " Oct 30 02:02:16.932507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.932490 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-openshift-service-ca\") pod \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " Oct 30 02:02:16.932631 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.932619 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-metrics-certs\") pod \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\" (UID: \"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47\") " Oct 30 02:02:16.932889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.932866 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-openshift-service-ca" pod "2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" (UID: "2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:16.933430 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.933408 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:16.935101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.935075 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-maas-api-tls" pod "2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" (UID: "2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:02:16.935189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.935135 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-kube-api-access-5cdrz" pod "2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" (UID: "2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47"). InnerVolumeSpecName "kube-api-access-5cdrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:16.935319 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:16.935297 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-metrics-certs" pod "2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" (UID: "2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:02:17.036072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.035245 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:17.036072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.035294 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cdrz\" (UniqueName: \"kubernetes.io/projected/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-kube-api-access-5cdrz\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:17.036072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.035309 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:17.296608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.294805 5035 generic.go:356] "Generic (PLEG): container finished" podID="2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" containerID="a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76" exitCode=0 Oct 30 02:02:17.296608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.294880 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" Oct 30 02:02:17.296608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.295971 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" event={"ID":"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47","Type":"ContainerDied","Data":"a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76"} Oct 30 02:02:17.296608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.296024 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn" event={"ID":"2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47","Type":"ContainerDied","Data":"c020fc616b89fa02325ad509a4d285c2d182a813bd33235618ec428438f936c0"} Oct 30 02:02:17.296608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.296053 5035 scope.go:122] "RemoveContainer" containerID="a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76" Oct 30 02:02:17.311488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.311466 5035 scope.go:122] "RemoveContainer" containerID="a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76" Oct 30 02:02:17.311929 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:02:17.311902 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76\": container with ID starting with a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76 not found: ID does not exist" containerID="a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76" Oct 30 02:02:17.312035 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.311942 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76"} err="failed to get container status \"a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76\": rpc error: code = NotFound desc = could not find container \"a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76\": container with ID starting with a774e34d601c58c4d0aabed0b24b875d4882db557ad1828fb25b77711266df76 not found: ID does not exist" Oct 30 02:02:17.354090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.354055 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn"] Oct 30 02:02:17.367229 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.367185 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-954773-7d44499fd-s25fn"] Oct 30 02:02:17.630840 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:02:17.630750 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:02:17.644493 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:17.644460 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" path="/var/lib/kubelet/pods/2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47/volumes" Oct 30 02:02:21.247144 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:21.247108 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:21.247730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:21.247194 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:23.264408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:23.264321 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:02:27.300428 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:27.300394 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:02:35.257410 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.257372 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q"] Oct 30 02:02:35.258350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.258327 5035 cpu_manager.go:414] "containerMap: removing container" podUID="2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" containerName="maas-api" Oct 30 02:02:35.258403 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.258353 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" containerName="maas-api" Oct 30 02:02:35.258558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.258545 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="2771cfbd-4b0e-4db1-9ea7-c11fc78b4c47" containerName="maas-api" Oct 30 02:02:35.262461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.262437 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.283923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.283879 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ea4bc0-serving-cert\"" Oct 30 02:02:35.290717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.290676 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q"] Oct 30 02:02:35.421601 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.421547 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-metrics-certs\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.421809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.421724 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thq2c\" (UniqueName: \"kubernetes.io/projected/af003945-c61f-4abe-b793-b3538cfc6d52-kube-api-access-thq2c\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.421873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.421834 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af003945-c61f-4abe-b793-b3538cfc6d52-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.421987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.421973 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.523247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.523141 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thq2c\" (UniqueName: \"kubernetes.io/projected/af003945-c61f-4abe-b793-b3538cfc6d52-kube-api-access-thq2c\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.523435 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.523252 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af003945-c61f-4abe-b793-b3538cfc6d52-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.523435 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.523390 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.523592 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.523449 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-metrics-certs\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.524067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.524041 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af003945-c61f-4abe-b793-b3538cfc6d52-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.526215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.526184 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.526351 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.526215 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-metrics-certs\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.548492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.548452 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thq2c\" (UniqueName: \"kubernetes.io/projected/af003945-c61f-4abe-b793-b3538cfc6d52-kube-api-access-thq2c\") pod \"maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.581633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.581595 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:35.904425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:35.904387 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q"] Oct 30 02:02:35.904966 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:02:35.904933 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf003945_c61f_4abe_b793_b3538cfc6d52.slice/crio-9f273d4ed9057e149e6f04a53ef208e6ec8ea08d2ddd1e5e5dc5f6865c68fb9c WatchSource:0}: Error finding container 9f273d4ed9057e149e6f04a53ef208e6ec8ea08d2ddd1e5e5dc5f6865c68fb9c: Status 404 returned error can't find the container with id 9f273d4ed9057e149e6f04a53ef208e6ec8ea08d2ddd1e5e5dc5f6865c68fb9c Oct 30 02:02:36.479173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:36.479134 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" event={"ID":"af003945-c61f-4abe-b793-b3538cfc6d52","Type":"ContainerStarted","Data":"2485b4cdd9ee5d813956e667844d249a66238b017b240fec9e88c3faa36f15c0"} Oct 30 02:02:36.479686 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:36.479224 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:36.479686 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:36.479242 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" event={"ID":"af003945-c61f-4abe-b793-b3538cfc6d52","Type":"ContainerStarted","Data":"9f273d4ed9057e149e6f04a53ef208e6ec8ea08d2ddd1e5e5dc5f6865c68fb9c"} Oct 30 02:02:36.526034 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:36.525922 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" podStartSLOduration=1.200689112 podStartE2EDuration="1.525903175s" totalImagesPullingTime="325.214063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:36.524220278 +0000 UTC m=+2035.027152498" watchObservedRunningTime="2026-10-30 02:02:36.525903175 +0000 UTC m=+2035.028835395" Oct 30 02:02:36.870390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:36.870295 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz"] Oct 30 02:02:36.870633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:36.870520 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" podUID="faf97867-abb8-41cc-8302-a6e9fce75769" containerName="maas-api" containerID="cri-o://cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850" gracePeriod=30 Oct 30 02:02:37.216943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.216911 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:02:37.381599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.370033 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-metrics-certs\") pod \"faf97867-abb8-41cc-8302-a6e9fce75769\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " Oct 30 02:02:37.381599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.370321 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-maas-api-tls\") pod \"faf97867-abb8-41cc-8302-a6e9fce75769\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " Oct 30 02:02:37.381599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.370376 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf97867-abb8-41cc-8302-a6e9fce75769-kube-api-access-p72xb\" (UniqueName: \"kubernetes.io/projected/faf97867-abb8-41cc-8302-a6e9fce75769-kube-api-access-p72xb\") pod \"faf97867-abb8-41cc-8302-a6e9fce75769\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " Oct 30 02:02:37.381599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.370441 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf97867-abb8-41cc-8302-a6e9fce75769-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf97867-abb8-41cc-8302-a6e9fce75769-openshift-service-ca\") pod \"faf97867-abb8-41cc-8302-a6e9fce75769\" (UID: \"faf97867-abb8-41cc-8302-a6e9fce75769\") " Oct 30 02:02:37.408558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.395990 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf97867-abb8-41cc-8302-a6e9fce75769-openshift-service-ca" pod "faf97867-abb8-41cc-8302-a6e9fce75769" (UID: "faf97867-abb8-41cc-8302-a6e9fce75769"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:37.445722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.443195 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf97867-abb8-41cc-8302-a6e9fce75769-kube-api-access-p72xb" pod "faf97867-abb8-41cc-8302-a6e9fce75769" (UID: "faf97867-abb8-41cc-8302-a6e9fce75769"). InnerVolumeSpecName "kube-api-access-p72xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:37.445722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.445510 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-metrics-certs" pod "faf97867-abb8-41cc-8302-a6e9fce75769" (UID: "faf97867-abb8-41cc-8302-a6e9fce75769"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:02:37.446855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.446813 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-maas-api-tls" pod "faf97867-abb8-41cc-8302-a6e9fce75769" (UID: "faf97867-abb8-41cc-8302-a6e9fce75769"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:02:37.473075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.473040 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:37.473075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.473075 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p72xb\" (UniqueName: \"kubernetes.io/projected/faf97867-abb8-41cc-8302-a6e9fce75769-kube-api-access-p72xb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:37.473304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.473087 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf97867-abb8-41cc-8302-a6e9fce75769-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:37.473304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.473098 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/faf97867-abb8-41cc-8302-a6e9fce75769-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:37.488626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.488586 5035 generic.go:356] "Generic (PLEG): container finished" podID="faf97867-abb8-41cc-8302-a6e9fce75769" containerID="cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850" exitCode=0 Oct 30 02:02:37.489112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.488648 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" Oct 30 02:02:37.489112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.488671 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" event={"ID":"faf97867-abb8-41cc-8302-a6e9fce75769","Type":"ContainerDied","Data":"cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850"} Oct 30 02:02:37.489112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.488751 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz" event={"ID":"faf97867-abb8-41cc-8302-a6e9fce75769","Type":"ContainerDied","Data":"1f36a08d89066cc4da5271949b0929b1c5f700009920e0086e23a430b352365a"} Oct 30 02:02:37.489112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.488776 5035 scope.go:122] "RemoveContainer" containerID="cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850" Oct 30 02:02:37.500841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.500818 5035 scope.go:122] "RemoveContainer" containerID="cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850" Oct 30 02:02:37.501168 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:02:37.501143 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850\": container with ID starting with cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850 not found: ID does not exist" containerID="cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850" Oct 30 02:02:37.501246 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.501175 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850"} err="failed to get container status \"cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850\": rpc error: code = NotFound desc = could not find container \"cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850\": container with ID starting with cd58db101b0a7f39381dddae4a823dc3a6a931957b47f984c4ef6d281036c850 not found: ID does not exist" Oct 30 02:02:37.547257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.547219 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz"] Oct 30 02:02:37.550962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.550925 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38bc0d-58797b497b-dh2dz"] Oct 30 02:02:37.651341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:37.651300 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf97867-abb8-41cc-8302-a6e9fce75769" path="/var/lib/kubelet/pods/faf97867-abb8-41cc-8302-a6e9fce75769/volumes" Oct 30 02:02:42.494730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:42.494690 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:02:47.118455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:47.118411 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc"] Oct 30 02:02:47.119273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:47.119235 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" containerID="cri-o://fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf" gracePeriod=30 Oct 30 02:02:47.632088 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:02:47.632046 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:02:50.435521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:50.435482 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n"] Oct 30 02:02:50.476289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:50.476250 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl"] Oct 30 02:02:52.425287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.424319 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:02:52.463329 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463283 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-data\") pod \"504f0ee4-4923-41e6-aa6f-7c976333058a\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " Oct 30 02:02:52.463606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463337 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-podinfo\") pod \"504f0ee4-4923-41e6-aa6f-7c976333058a\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " Oct 30 02:02:52.463606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463387 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-certs\") pod \"504f0ee4-4923-41e6-aa6f-7c976333058a\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " Oct 30 02:02:52.463606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463518 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-envoy\") pod \"504f0ee4-4923-41e6-aa6f-7c976333058a\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " Oct 30 02:02:52.463606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463567 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-credential-socket\") pod \"504f0ee4-4923-41e6-aa6f-7c976333058a\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " Oct 30 02:02:52.463606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463562 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-data" pod "504f0ee4-4923-41e6-aa6f-7c976333058a" (UID: "504f0ee4-4923-41e6-aa6f-7c976333058a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:52.463938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463635 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-token\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-token\") pod \"504f0ee4-4923-41e6-aa6f-7c976333058a\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " Oct 30 02:02:52.463938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463702 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-socket\") pod \"504f0ee4-4923-41e6-aa6f-7c976333058a\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " Oct 30 02:02:52.463938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463711 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-certs" pod "504f0ee4-4923-41e6-aa6f-7c976333058a" (UID: "504f0ee4-4923-41e6-aa6f-7c976333058a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:52.463938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463793 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-credential-socket" pod "504f0ee4-4923-41e6-aa6f-7c976333058a" (UID: "504f0ee4-4923-41e6-aa6f-7c976333058a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:52.463938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463900 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-kube-api-access-zr4wq\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-kube-api-access-zr4wq\") pod \"504f0ee4-4923-41e6-aa6f-7c976333058a\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " Oct 30 02:02:52.463938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463921 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-socket" pod "504f0ee4-4923-41e6-aa6f-7c976333058a" (UID: "504f0ee4-4923-41e6-aa6f-7c976333058a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:52.464210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.463941 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504f0ee4-4923-41e6-aa6f-7c976333058a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504f0ee4-4923-41e6-aa6f-7c976333058a-istiod-ca-cert\") pod \"504f0ee4-4923-41e6-aa6f-7c976333058a\" (UID: \"504f0ee4-4923-41e6-aa6f-7c976333058a\") " Oct 30 02:02:52.464304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.464281 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504f0ee4-4923-41e6-aa6f-7c976333058a-istiod-ca-cert" pod "504f0ee4-4923-41e6-aa6f-7c976333058a" (UID: "504f0ee4-4923-41e6-aa6f-7c976333058a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:52.464719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.464692 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:52.464836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.464721 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:52.464836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.464738 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/504f0ee4-4923-41e6-aa6f-7c976333058a-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:52.464836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.464754 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:52.464836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.464767 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:52.466522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.466487 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-kube-api-access-zr4wq" pod "504f0ee4-4923-41e6-aa6f-7c976333058a" (UID: "504f0ee4-4923-41e6-aa6f-7c976333058a"). InnerVolumeSpecName "kube-api-access-zr4wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:52.467244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.467215 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-podinfo" pod "504f0ee4-4923-41e6-aa6f-7c976333058a" (UID: "504f0ee4-4923-41e6-aa6f-7c976333058a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:52.467373 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.467354 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-token" pod "504f0ee4-4923-41e6-aa6f-7c976333058a" (UID: "504f0ee4-4923-41e6-aa6f-7c976333058a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:52.467656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.467632 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-envoy" pod "504f0ee4-4923-41e6-aa6f-7c976333058a" (UID: "504f0ee4-4923-41e6-aa6f-7c976333058a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:52.566597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.566501 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr4wq\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-kube-api-access-zr4wq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:52.566597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.566597 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:52.566597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.566609 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:52.566865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.566617 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/504f0ee4-4923-41e6-aa6f-7c976333058a-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:52.610330 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.610290 5035 generic.go:356] "Generic (PLEG): container finished" podID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerID="fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf" exitCode=0 Oct 30 02:02:52.610510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.610344 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" Oct 30 02:02:52.610510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.610366 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" event={"ID":"504f0ee4-4923-41e6-aa6f-7c976333058a","Type":"ContainerDied","Data":"fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf"} Oct 30 02:02:52.610510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.610443 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc" event={"ID":"504f0ee4-4923-41e6-aa6f-7c976333058a","Type":"ContainerDied","Data":"e91f08fc8b5a3421f4d02fd4b96a346ea2b0b65c897ce342317084520eb1f1ab"} Oct 30 02:02:52.610510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.610466 5035 scope.go:122] "RemoveContainer" containerID="fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf" Oct 30 02:02:52.623077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.623049 5035 scope.go:122] "RemoveContainer" containerID="d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3" Oct 30 02:02:52.634324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.634297 5035 scope.go:122] "RemoveContainer" containerID="fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf" Oct 30 02:02:52.634702 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:02:52.634669 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf\": container with ID starting with fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf not found: ID does not exist" containerID="fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf" Oct 30 02:02:52.634841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.634705 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf"} err="failed to get container status \"fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf\": rpc error: code = NotFound desc = could not find container \"fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf\": container with ID starting with fca655b60b084b7b5afe14d5286cc95c8f44cfb3ae66ae78785d74d05daaebdf not found: ID does not exist" Oct 30 02:02:52.634841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.634726 5035 scope.go:122] "RemoveContainer" containerID="d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3" Oct 30 02:02:52.635025 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:02:52.635003 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3\": container with ID starting with d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3 not found: ID does not exist" containerID="d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3" Oct 30 02:02:52.635078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.635032 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3"} err="failed to get container status \"d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3\": rpc error: code = NotFound desc = could not find container \"d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3\": container with ID starting with d763c10d184c25a7cddf5fa0eb9b262037c50ca5d827994cc4dd4926b8d28fa3 not found: ID does not exist" Oct 30 02:02:52.668730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.668623 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc"] Oct 30 02:02:52.672864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:52.672832 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-38bc0d-openshift-default-58c74bbb77-b7csc"] Oct 30 02:02:53.644638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:53.644599 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" path="/var/lib/kubelet/pods/504f0ee4-4923-41e6-aa6f-7c976333058a/volumes" Oct 30 02:02:55.042732 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.042687 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j"] Oct 30 02:02:55.043381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.043366 5035 cpu_manager.go:414] "containerMap: removing container" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" Oct 30 02:02:55.043427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.043382 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" Oct 30 02:02:55.043427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.043399 5035 cpu_manager.go:414] "containerMap: removing container" podUID="faf97867-abb8-41cc-8302-a6e9fce75769" containerName="maas-api" Oct 30 02:02:55.043427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.043404 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf97867-abb8-41cc-8302-a6e9fce75769" containerName="maas-api" Oct 30 02:02:55.043427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.043414 5035 cpu_manager.go:414] "containerMap: removing container" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" Oct 30 02:02:55.043427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.043420 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" Oct 30 02:02:55.043606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.043553 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" Oct 30 02:02:55.043606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.043563 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf97867-abb8-41cc-8302-a6e9fce75769" containerName="maas-api" Oct 30 02:02:55.043606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.043569 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="504f0ee4-4923-41e6-aa6f-7c976333058a" containerName="istio-proxy" Oct 30 02:02:55.048776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.048752 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:02:55.066819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.066784 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j"] Oct 30 02:02:55.075641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.075606 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-2979be\"" Oct 30 02:02:55.075864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.075841 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-2979be-dockercfg-8cx86\"" Oct 30 02:02:55.251556 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.236443 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j\" (UID: \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:02:55.251556 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.236499 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gg77\" (UniqueName: \"kubernetes.io/projected/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-kube-api-access-9gg77\") pod \"payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j\" (UID: \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:02:55.338257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.338146 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j\" (UID: \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:02:55.338257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.338206 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gg77\" (UniqueName: \"kubernetes.io/projected/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-kube-api-access-9gg77\") pod \"payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j\" (UID: \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:02:55.339030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.339002 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j\" (UID: \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:02:55.362974 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.362941 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gg77\" (UniqueName: \"kubernetes.io/projected/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-kube-api-access-9gg77\") pod \"payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j\" (UID: \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:02:55.364793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.364769 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:02:55.449211 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.447593 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" podUID="c74405b1-8c63-4204-8c23-4d8540e7bdc4" containerName="payload-processing" containerID="cri-o://b8a870c442f1c3e909a5ce185c79efb713b07014334e73be1db3ca8074c4c025" gracePeriod=25 Oct 30 02:02:55.477240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.477199 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" podUID="b68152d0-8586-49d1-be82-1c5f8887936a" containerName="payload-pre-processing" containerID="cri-o://d9f03ec2970db4a803af19ea4e5c1fd2c6abe6122fdab249eea7237a42b95d92" gracePeriod=25 Oct 30 02:02:55.639466 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.639432 5035 generic.go:356] "Generic (PLEG): container finished" podID="c74405b1-8c63-4204-8c23-4d8540e7bdc4" containerID="b8a870c442f1c3e909a5ce185c79efb713b07014334e73be1db3ca8074c4c025" exitCode=0 Oct 30 02:02:55.643850 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.643817 5035 generic.go:356] "Generic (PLEG): container finished" podID="b68152d0-8586-49d1-be82-1c5f8887936a" containerID="d9f03ec2970db4a803af19ea4e5c1fd2c6abe6122fdab249eea7237a42b95d92" exitCode=0 Oct 30 02:02:55.648593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.648553 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" event={"ID":"c74405b1-8c63-4204-8c23-4d8540e7bdc4","Type":"ContainerDied","Data":"b8a870c442f1c3e909a5ce185c79efb713b07014334e73be1db3ca8074c4c025"} Oct 30 02:02:55.648782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.648626 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" event={"ID":"b68152d0-8586-49d1-be82-1c5f8887936a","Type":"ContainerDied","Data":"d9f03ec2970db4a803af19ea4e5c1fd2c6abe6122fdab249eea7237a42b95d92"} Oct 30 02:02:55.688442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.688270 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j"] Oct 30 02:02:55.767096 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.767061 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:55.822178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.822148 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:55.852604 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.852486 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74405b1-8c63-4204-8c23-4d8540e7bdc4-kube-api-access-qqpxv\" (UniqueName: \"kubernetes.io/projected/c74405b1-8c63-4204-8c23-4d8540e7bdc4-kube-api-access-qqpxv\") pod \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\" (UID: \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\") " Oct 30 02:02:55.852799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.852744 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68152d0-8586-49d1-be82-1c5f8887936a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68152d0-8586-49d1-be82-1c5f8887936a-plugins-config-volume\") pod \"b68152d0-8586-49d1-be82-1c5f8887936a\" (UID: \"b68152d0-8586-49d1-be82-1c5f8887936a\") " Oct 30 02:02:55.852876 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.852862 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74405b1-8c63-4204-8c23-4d8540e7bdc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74405b1-8c63-4204-8c23-4d8540e7bdc4-plugins-config-volume\") pod \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\" (UID: \"c74405b1-8c63-4204-8c23-4d8540e7bdc4\") " Oct 30 02:02:55.852934 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.852913 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68152d0-8586-49d1-be82-1c5f8887936a-kube-api-access-84vwb\" (UniqueName: \"kubernetes.io/projected/b68152d0-8586-49d1-be82-1c5f8887936a-kube-api-access-84vwb\") pod \"b68152d0-8586-49d1-be82-1c5f8887936a\" (UID: \"b68152d0-8586-49d1-be82-1c5f8887936a\") " Oct 30 02:02:55.853431 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.853387 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68152d0-8586-49d1-be82-1c5f8887936a-plugins-config-volume" pod "b68152d0-8586-49d1-be82-1c5f8887936a" (UID: "b68152d0-8586-49d1-be82-1c5f8887936a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:55.853431 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.853387 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74405b1-8c63-4204-8c23-4d8540e7bdc4-plugins-config-volume" pod "c74405b1-8c63-4204-8c23-4d8540e7bdc4" (UID: "c74405b1-8c63-4204-8c23-4d8540e7bdc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:55.853650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.853599 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68152d0-8586-49d1-be82-1c5f8887936a-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:55.853650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.853623 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74405b1-8c63-4204-8c23-4d8540e7bdc4-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:55.855504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.855474 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74405b1-8c63-4204-8c23-4d8540e7bdc4-kube-api-access-qqpxv" pod "c74405b1-8c63-4204-8c23-4d8540e7bdc4" (UID: "c74405b1-8c63-4204-8c23-4d8540e7bdc4"). InnerVolumeSpecName "kube-api-access-qqpxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:55.855659 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.855634 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68152d0-8586-49d1-be82-1c5f8887936a-kube-api-access-84vwb" pod "b68152d0-8586-49d1-be82-1c5f8887936a" (UID: "b68152d0-8586-49d1-be82-1c5f8887936a"). InnerVolumeSpecName "kube-api-access-84vwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:55.955461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.955416 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84vwb\" (UniqueName: \"kubernetes.io/projected/b68152d0-8586-49d1-be82-1c5f8887936a-kube-api-access-84vwb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:55.955461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:55.955457 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqpxv\" (UniqueName: \"kubernetes.io/projected/c74405b1-8c63-4204-8c23-4d8540e7bdc4-kube-api-access-qqpxv\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:02:56.667059 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.666956 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" Oct 30 02:02:56.667059 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.667040 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl" event={"ID":"b68152d0-8586-49d1-be82-1c5f8887936a","Type":"ContainerDied","Data":"0e2060847719c98c3a9e6bfc1c42bb608463ff9c79faa591b012502a6710db75"} Oct 30 02:02:56.667628 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.667107 5035 scope.go:122] "RemoveContainer" containerID="d9f03ec2970db4a803af19ea4e5c1fd2c6abe6122fdab249eea7237a42b95d92" Oct 30 02:02:56.670190 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.670157 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" event={"ID":"e59d5a6e-c78e-4d7b-8edb-8b014303aad7","Type":"ContainerStarted","Data":"f568f2d2244455df186aee27c2d84ea6a5bc9891841dd9707334faf7d1d8b2db"} Oct 30 02:02:56.670325 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.670270 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" event={"ID":"e59d5a6e-c78e-4d7b-8edb-8b014303aad7","Type":"ContainerStarted","Data":"8345f5cd1919347ced793fef574004821c7795c2fbdd4db645b81c17f308b6e6"} Oct 30 02:02:56.671076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.671053 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:02:56.673686 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.673651 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" event={"ID":"c74405b1-8c63-4204-8c23-4d8540e7bdc4","Type":"ContainerDied","Data":"c6a13b580e7e1b4c66a62b40c7345c4b42307ae39b3edab26b7f3ce0ab7e2106"} Oct 30 02:02:56.673849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.673782 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n" Oct 30 02:02:56.684476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.684201 5035 scope.go:122] "RemoveContainer" containerID="b8a870c442f1c3e909a5ce185c79efb713b07014334e73be1db3ca8074c4c025" Oct 30 02:02:56.746851 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.746787 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" podStartSLOduration=1.746768131 podStartE2EDuration="1.746768131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:56.742895764 +0000 UTC m=+2055.245827985" watchObservedRunningTime="2026-10-30 02:02:56.746768131 +0000 UTC m=+2055.249700352" Oct 30 02:02:56.783273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.783225 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl"] Oct 30 02:02:56.786889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.786857 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-57776d77b7-zzmrl"] Oct 30 02:02:56.823579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.823543 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n"] Oct 30 02:02:56.830563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:56.830509 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-168989-5cfc8db7cd-t522n"] Oct 30 02:02:57.644545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:57.644496 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68152d0-8586-49d1-be82-1c5f8887936a" path="/var/lib/kubelet/pods/b68152d0-8586-49d1-be82-1c5f8887936a/volumes" Oct 30 02:02:57.645141 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:02:57.645118 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74405b1-8c63-4204-8c23-4d8540e7bdc4" path="/var/lib/kubelet/pods/c74405b1-8c63-4204-8c23-4d8540e7bdc4/volumes" Oct 30 02:03:00.047792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.047742 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm"] Oct 30 02:03:00.049274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.049245 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c74405b1-8c63-4204-8c23-4d8540e7bdc4" containerName="payload-processing" Oct 30 02:03:00.049274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.049276 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74405b1-8c63-4204-8c23-4d8540e7bdc4" containerName="payload-processing" Oct 30 02:03:00.049500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.049307 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b68152d0-8586-49d1-be82-1c5f8887936a" containerName="payload-pre-processing" Oct 30 02:03:00.049500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.049316 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68152d0-8586-49d1-be82-1c5f8887936a" containerName="payload-pre-processing" Oct 30 02:03:00.049735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.049718 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74405b1-8c63-4204-8c23-4d8540e7bdc4" containerName="payload-processing" Oct 30 02:03:00.049818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.049747 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68152d0-8586-49d1-be82-1c5f8887936a" containerName="payload-pre-processing" Oct 30 02:03:00.059582 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.059505 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.085379 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.085341 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-003ec6-openshift-default-dockercfg-ggl7h\"" Oct 30 02:03:00.151099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.151057 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm"] Oct 30 02:03:00.275552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.275438 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-token\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.275552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.275499 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-certs\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.275867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.275567 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-credential-socket\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.275867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.275596 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-data\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.275867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.275627 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-podinfo\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.275867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.275657 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istiod-ca-cert\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.275867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.275712 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2ccg\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-kube-api-access-t2ccg\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.275867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.275798 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-envoy\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.275867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.275849 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-socket\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.385615 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.385057 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-envoy\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.385615 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.385169 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-socket\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.385615 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.385576 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-token\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.385939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.385633 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-certs\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.385939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.385698 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-credential-socket\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.385939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.385729 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-data\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.385939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.385774 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-podinfo\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.385939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.385808 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istiod-ca-cert\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.385939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.385884 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2ccg\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-kube-api-access-t2ccg\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.412616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.391375 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-credential-socket\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.412616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.391691 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-socket\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.412616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.392275 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-certs\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.412616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.394478 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-data\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.412616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.394782 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-envoy\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.412616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.395449 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istiod-ca-cert\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.469568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.459401 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-podinfo\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.500590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.500501 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2ccg\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-kube-api-access-t2ccg\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.534551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.529217 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-token\") pod \"e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.535992 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.535950 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:00.951361 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.951322 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm"] Oct 30 02:03:00.953615 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:03:00.953582 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bdac7d_3ef7_4906_ac0f_57f33ec5a08c.slice/crio-9e89d1806fe89b422abd622f810b8869bb531e4bef186d28b57dfd72f9f2f7fc WatchSource:0}: Error finding container 9e89d1806fe89b422abd622f810b8869bb531e4bef186d28b57dfd72f9f2f7fc: Status 404 returned error can't find the container with id 9e89d1806fe89b422abd622f810b8869bb531e4bef186d28b57dfd72f9f2f7fc Oct 30 02:03:00.956561 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.956501 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:03:00.956681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.956601 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:03:00.956681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:00.956633 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:03:01.786554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:01.775819 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" event={"ID":"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c","Type":"ContainerStarted","Data":"a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0"} Oct 30 02:03:01.786554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:01.775915 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" event={"ID":"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c","Type":"ContainerStarted","Data":"9e89d1806fe89b422abd622f810b8869bb531e4bef186d28b57dfd72f9f2f7fc"} Oct 30 02:03:01.842782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:01.842705 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" podStartSLOduration=2.8426850569999997 podStartE2EDuration="2.842685057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:01.838668194 +0000 UTC m=+2060.341600411" watchObservedRunningTime="2026-10-30 02:03:01.842685057 +0000 UTC m=+2060.345617276" Oct 30 02:03:02.537225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:02.537179 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:02.543257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:02.543224 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:02.821692 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:02.821498 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:02.822850 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:02.822829 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:03:07.133789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.133721 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:03:07.134392 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.133799 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:03:07.866028 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.865990 5035 generic.go:356] "Generic (PLEG): container finished" podID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerID="965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67" exitCode=0 Oct 30 02:03:07.866233 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.866058 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" event={"ID":"1b81a0ab-3b26-403b-bfb3-b36cf060240f","Type":"ContainerDied","Data":"965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67"} Oct 30 02:03:07.866678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.866656 5035 scope.go:122] "RemoveContainer" containerID="965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67" Oct 30 02:03:07.868538 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.868506 5035 generic.go:356] "Generic (PLEG): container finished" podID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerID="348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84" exitCode=0 Oct 30 02:03:07.868727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.868711 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" event={"ID":"f5fb784e-0d01-4def-bdc1-63259c9e1fbf","Type":"ContainerDied","Data":"348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84"} Oct 30 02:03:07.868772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.868723 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:03:07.868858 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.868810 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:03:07.868900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.868861 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:03:07.869159 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.869142 5035 scope.go:122] "RemoveContainer" containerID="348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84" Oct 30 02:03:07.871662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.871630 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:03:07.871799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.871722 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:03:07.871799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:07.871772 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:03:08.730947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:08.730917 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:03:08.935238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:08.935200 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" event={"ID":"1b81a0ab-3b26-403b-bfb3-b36cf060240f","Type":"ContainerStarted","Data":"f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0"} Oct 30 02:03:08.938033 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:08.938001 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" event={"ID":"f5fb784e-0d01-4def-bdc1-63259c9e1fbf","Type":"ContainerStarted","Data":"2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3"} Oct 30 02:03:08.950458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:08.950425 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:03:09.096619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:09.096033 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:03:09.096619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:09.096183 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 02:03:09.096619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:09.096230 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 02:03:09.952274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:09.952233 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 02:03:09.952768 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:09.952308 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 02:03:10.093086 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:10.093047 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 02:03:10.093389 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:10.093356 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 02:03:10.950520 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:10.950483 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 02:03:10.950814 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:10.950789 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 02:03:11.093983 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:11.093941 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 02:03:11.094600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:11.094563 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 02:03:11.244173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:11.244082 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:03:11.950237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:11.950201 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 02:03:11.950510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:11.950476 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 02:03:12.108560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:12.100054 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 02:03:12.108560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:12.100145 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 02:03:12.949822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:12.949785 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 02:03:12.950135 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:12.950103 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 02:03:13.095052 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:13.095007 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 02:03:13.095382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:13.095346 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 02:03:13.950331 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:13.950289 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 02:03:13.950822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:13.950356 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 02:03:14.096013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:14.095970 5035 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 02:03:14.096335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:14.096302 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 02:03:14.954920 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:14.954885 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:03:14.956175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:14.956141 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:03:15.099035 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:15.099002 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:03:15.099485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:15.099456 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:03:15.100310 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:15.100281 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:03:15.990645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:15.990609 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr"] Oct 30 02:03:15.995171 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:15.995143 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:16.020760 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.020721 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr"] Oct 30 02:03:16.029746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.029711 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-003ec6\"" Oct 30 02:03:16.029970 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.029837 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-003ec6-dockercfg-7nlxr\"" Oct 30 02:03:16.056031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.055997 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27"] Oct 30 02:03:16.057597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.057555 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq6wg\" (UniqueName: \"kubernetes.io/projected/40658182-48e2-4357-83a9-a66778baa6b8-kube-api-access-rq6wg\") pod \"payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr\" (UID: \"40658182-48e2-4357-83a9-a66778baa6b8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:16.057746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.057658 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40658182-48e2-4357-83a9-a66778baa6b8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr\" (UID: \"40658182-48e2-4357-83a9-a66778baa6b8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:16.059739 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.059717 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:16.078920 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.078891 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27"] Oct 30 02:03:16.224555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.219235 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40658182-48e2-4357-83a9-a66778baa6b8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr\" (UID: \"40658182-48e2-4357-83a9-a66778baa6b8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:16.224555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.219469 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27\" (UID: \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:16.224555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.219603 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jlh\" (UniqueName: \"kubernetes.io/projected/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-kube-api-access-g7jlh\") pod \"payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27\" (UID: \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:16.224555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.219768 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq6wg\" (UniqueName: \"kubernetes.io/projected/40658182-48e2-4357-83a9-a66778baa6b8-kube-api-access-rq6wg\") pod \"payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr\" (UID: \"40658182-48e2-4357-83a9-a66778baa6b8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:16.224555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.221024 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40658182-48e2-4357-83a9-a66778baa6b8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr\" (UID: \"40658182-48e2-4357-83a9-a66778baa6b8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:16.262839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.257712 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq6wg\" (UniqueName: \"kubernetes.io/projected/40658182-48e2-4357-83a9-a66778baa6b8-kube-api-access-rq6wg\") pod \"payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr\" (UID: \"40658182-48e2-4357-83a9-a66778baa6b8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:16.310638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.310599 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:16.321716 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.321671 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27\" (UID: \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:16.321899 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.321768 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jlh\" (UniqueName: \"kubernetes.io/projected/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-kube-api-access-g7jlh\") pod \"payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27\" (UID: \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:16.322405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.322381 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27\" (UID: \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:16.360918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.360883 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jlh\" (UniqueName: \"kubernetes.io/projected/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-kube-api-access-g7jlh\") pod \"payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27\" (UID: \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:16.378782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.378554 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:16.560469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.560376 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm"] Oct 30 02:03:16.564773 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.564743 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.588230 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.588193 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm"] Oct 30 02:03:16.602876 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.602834 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-2979be-serving-cert\"" Oct 30 02:03:16.638149 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.638111 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr"] Oct 30 02:03:16.639393 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:03:16.639363 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40658182_48e2_4357_83a9_a66778baa6b8.slice/crio-ee2f57031d788adfccf887bbca4d0fca1197c1aa2be15bb0796f588b061cdfe6 WatchSource:0}: Error finding container ee2f57031d788adfccf887bbca4d0fca1197c1aa2be15bb0796f588b061cdfe6: Status 404 returned error can't find the container with id ee2f57031d788adfccf887bbca4d0fca1197c1aa2be15bb0796f588b061cdfe6 Oct 30 02:03:16.727847 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.718467 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27"] Oct 30 02:03:16.752554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.750637 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6qwt\" (UniqueName: \"kubernetes.io/projected/de6ff5cf-766f-49c0-b962-305c86af4280-kube-api-access-b6qwt\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.752554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.750719 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-metrics-certs\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.752554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.750811 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.752554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.750840 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6ff5cf-766f-49c0-b962-305c86af4280-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.852738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.852706 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6qwt\" (UniqueName: \"kubernetes.io/projected/de6ff5cf-766f-49c0-b962-305c86af4280-kube-api-access-b6qwt\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.852883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.852753 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-metrics-certs\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.852883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.852804 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.852883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.852825 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6ff5cf-766f-49c0-b962-305c86af4280-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.853040 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:03:16.852919 5035 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-serving-cert: secret "maas-api-e2e-praxis-a-2979be-serving-cert" not found Oct 30 02:03:16.853040 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:03:16.852982 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls podName:de6ff5cf-766f-49c0-b962-305c86af4280 nodeName:}" failed. No retries permitted until 2026-10-30 02:03:17.352961899 +0000 UTC m=+2075.855894114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls") pod "maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" (UID: "de6ff5cf-766f-49c0-b962-305c86af4280") : secret "maas-api-e2e-praxis-a-2979be-serving-cert" not found Oct 30 02:03:16.853697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.853499 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6ff5cf-766f-49c0-b962-305c86af4280-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.855173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.855152 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-metrics-certs\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:16.877030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:16.877000 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6qwt\" (UniqueName: \"kubernetes.io/projected/de6ff5cf-766f-49c0-b962-305c86af4280-kube-api-access-b6qwt\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:17.056153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.056115 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" event={"ID":"0c6bcae2-5da9-40ce-9f94-def87a42ac3f","Type":"ContainerStarted","Data":"a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36"} Oct 30 02:03:17.056675 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.056173 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" event={"ID":"0c6bcae2-5da9-40ce-9f94-def87a42ac3f","Type":"ContainerStarted","Data":"7f734409bc68c5ae5d3127af1a5cf2a3aa70ded975f5ca5300fe88dd92439c03"} Oct 30 02:03:17.056675 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.056201 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:17.058429 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.058392 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" event={"ID":"40658182-48e2-4357-83a9-a66778baa6b8","Type":"ContainerStarted","Data":"4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5"} Oct 30 02:03:17.058606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.058451 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" event={"ID":"40658182-48e2-4357-83a9-a66778baa6b8","Type":"ContainerStarted","Data":"ee2f57031d788adfccf887bbca4d0fca1197c1aa2be15bb0796f588b061cdfe6"} Oct 30 02:03:17.058606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.058557 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:17.110082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.109956 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" podStartSLOduration=1.109933682 podStartE2EDuration="1.109933682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:17.108828628 +0000 UTC m=+2075.611760850" watchObservedRunningTime="2026-10-30 02:03:17.109933682 +0000 UTC m=+2075.612865904" Oct 30 02:03:17.165097 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.165041 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" podStartSLOduration=2.165027712 podStartE2EDuration="2.165027712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:17.160443115 +0000 UTC m=+2075.663375336" watchObservedRunningTime="2026-10-30 02:03:17.165027712 +0000 UTC m=+2075.667959931" Oct 30 02:03:17.360916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.360821 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:17.363619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.363589 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:17.486692 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.486652 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:17.637897 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:03:17.637792 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:03:17.829019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:17.828984 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm"] Oct 30 02:03:17.829770 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:03:17.829742 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6ff5cf_766f_49c0_b962_305c86af4280.slice/crio-13deb363a0832853903c3f280726afceadbbc805555e7747f84282843779be5d WatchSource:0}: Error finding container 13deb363a0832853903c3f280726afceadbbc805555e7747f84282843779be5d: Status 404 returned error can't find the container with id 13deb363a0832853903c3f280726afceadbbc805555e7747f84282843779be5d Oct 30 02:03:18.068354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:18.068302 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" event={"ID":"de6ff5cf-766f-49c0-b962-305c86af4280","Type":"ContainerStarted","Data":"13deb363a0832853903c3f280726afceadbbc805555e7747f84282843779be5d"} Oct 30 02:03:19.078870 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:19.078813 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" event={"ID":"de6ff5cf-766f-49c0-b962-305c86af4280","Type":"ContainerStarted","Data":"3e5a3d326ec9a4679d8a29ec621a8d895bc279190804ccb4996c7c687e4a9240"} Oct 30 02:03:19.079346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:19.078997 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:19.128469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:19.128379 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" podStartSLOduration=2.646186405 podStartE2EDuration="3.128350863s" totalImagesPullingTime="482.164458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:19.127774971 +0000 UTC m=+2077.630707185" watchObservedRunningTime="2026-10-30 02:03:19.128350863 +0000 UTC m=+2077.631283081" Oct 30 02:03:25.094704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:25.094671 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:03:28.072237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:28.072201 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:03:28.072732 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:28.072291 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:03:34.345562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.345510 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c"] Oct 30 02:03:34.350207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.350172 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:34.375134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.375093 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c"] Oct 30 02:03:34.384294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.384253 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0f3cd8\"" Oct 30 02:03:34.384917 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.384066 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0f3cd8-dockercfg-jrr4c\"" Oct 30 02:03:34.424407 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.424365 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mk2z\" (UniqueName: \"kubernetes.io/projected/c58096a8-ee94-4650-b166-ff870a6b5f08-kube-api-access-5mk2z\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c\" (UID: \"c58096a8-ee94-4650-b166-ff870a6b5f08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:34.424651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.424623 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58096a8-ee94-4650-b166-ff870a6b5f08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c\" (UID: \"c58096a8-ee94-4650-b166-ff870a6b5f08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:34.603021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.602908 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58096a8-ee94-4650-b166-ff870a6b5f08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c\" (UID: \"c58096a8-ee94-4650-b166-ff870a6b5f08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:34.603215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.603178 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mk2z\" (UniqueName: \"kubernetes.io/projected/c58096a8-ee94-4650-b166-ff870a6b5f08-kube-api-access-5mk2z\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c\" (UID: \"c58096a8-ee94-4650-b166-ff870a6b5f08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:34.639643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.621710 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58096a8-ee94-4650-b166-ff870a6b5f08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c\" (UID: \"c58096a8-ee94-4650-b166-ff870a6b5f08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:34.639643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.636320 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mk2z\" (UniqueName: \"kubernetes.io/projected/c58096a8-ee94-4650-b166-ff870a6b5f08-kube-api-access-5mk2z\") pod \"payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c\" (UID: \"c58096a8-ee94-4650-b166-ff870a6b5f08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:34.664808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.664770 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:34.980401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:34.980365 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c"] Oct 30 02:03:34.981093 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:03:34.981064 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58096a8_ee94_4650_b166_ff870a6b5f08.slice/crio-365dc008bc02617d9f85211e16d2ae27ba9a691f91bca685eee223bb101617fa WatchSource:0}: Error finding container 365dc008bc02617d9f85211e16d2ae27ba9a691f91bca685eee223bb101617fa: Status 404 returned error can't find the container with id 365dc008bc02617d9f85211e16d2ae27ba9a691f91bca685eee223bb101617fa Oct 30 02:03:35.243353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:35.243253 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" event={"ID":"c58096a8-ee94-4650-b166-ff870a6b5f08","Type":"ContainerStarted","Data":"09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988"} Oct 30 02:03:35.243353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:35.243318 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:35.243353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:35.243335 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" event={"ID":"c58096a8-ee94-4650-b166-ff870a6b5f08","Type":"ContainerStarted","Data":"365dc008bc02617d9f85211e16d2ae27ba9a691f91bca685eee223bb101617fa"} Oct 30 02:03:35.291677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:35.291610 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" podStartSLOduration=1.291596452 podStartE2EDuration="1.291596452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:35.290694371 +0000 UTC m=+2093.793626591" watchObservedRunningTime="2026-10-30 02:03:35.291596452 +0000 UTC m=+2093.794528672" Oct 30 02:03:43.670181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:43.670078 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:03:43.686659 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:43.686632 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:03:43.704340 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:43.704310 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:03:43.720789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:43.720760 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:03:46.257607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:46.257570 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:03:47.631326 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:03:47.631264 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:03:47.748681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:47.748644 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h"] Oct 30 02:03:47.748906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:47.748880 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" containerID="cri-o://f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62" gracePeriod=30 Oct 30 02:03:53.070309 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.070276 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.195700 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-kube-api-access-hq7m4\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-kube-api-access-hq7m4\") pod \"2ae724a0-57d2-4116-9537-31e7aa2aea87\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.195864 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-certs\") pod \"2ae724a0-57d2-4116-9537-31e7aa2aea87\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.195948 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-envoy\") pod \"2ae724a0-57d2-4116-9537-31e7aa2aea87\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.196064 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-credential-socket\") pod \"2ae724a0-57d2-4116-9537-31e7aa2aea87\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.196106 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-podinfo\") pod \"2ae724a0-57d2-4116-9537-31e7aa2aea87\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.196187 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-token\") pod \"2ae724a0-57d2-4116-9537-31e7aa2aea87\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.196237 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-data\") pod \"2ae724a0-57d2-4116-9537-31e7aa2aea87\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.196276 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-socket\") pod \"2ae724a0-57d2-4116-9537-31e7aa2aea87\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.196380 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae724a0-57d2-4116-9537-31e7aa2aea87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae724a0-57d2-4116-9537-31e7aa2aea87-istiod-ca-cert\") pod \"2ae724a0-57d2-4116-9537-31e7aa2aea87\" (UID: \"2ae724a0-57d2-4116-9537-31e7aa2aea87\") " Oct 30 02:03:53.197903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.197583 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-credential-socket" pod "2ae724a0-57d2-4116-9537-31e7aa2aea87" (UID: "2ae724a0-57d2-4116-9537-31e7aa2aea87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:03:53.198668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.197964 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:03:53.198668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.198432 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-data" pod "2ae724a0-57d2-4116-9537-31e7aa2aea87" (UID: "2ae724a0-57d2-4116-9537-31e7aa2aea87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:03:53.199077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.198944 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae724a0-57d2-4116-9537-31e7aa2aea87-istiod-ca-cert" pod "2ae724a0-57d2-4116-9537-31e7aa2aea87" (UID: "2ae724a0-57d2-4116-9537-31e7aa2aea87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:03:53.199348 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.199195 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-socket" pod "2ae724a0-57d2-4116-9537-31e7aa2aea87" (UID: "2ae724a0-57d2-4116-9537-31e7aa2aea87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:03:53.199509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.199488 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-certs" pod "2ae724a0-57d2-4116-9537-31e7aa2aea87" (UID: "2ae724a0-57d2-4116-9537-31e7aa2aea87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:03:53.200545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.200487 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-kube-api-access-hq7m4" pod "2ae724a0-57d2-4116-9537-31e7aa2aea87" (UID: "2ae724a0-57d2-4116-9537-31e7aa2aea87"). InnerVolumeSpecName "kube-api-access-hq7m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:03:53.200875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.200848 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-podinfo" pod "2ae724a0-57d2-4116-9537-31e7aa2aea87" (UID: "2ae724a0-57d2-4116-9537-31e7aa2aea87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:03:53.202156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.202128 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-envoy" pod "2ae724a0-57d2-4116-9537-31e7aa2aea87" (UID: "2ae724a0-57d2-4116-9537-31e7aa2aea87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:03:53.202256 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.202229 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-token" pod "2ae724a0-57d2-4116-9537-31e7aa2aea87" (UID: "2ae724a0-57d2-4116-9537-31e7aa2aea87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:03:53.299661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.299515 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:03:53.299661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.299603 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:03:53.299661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.299620 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae724a0-57d2-4116-9537-31e7aa2aea87-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:03:53.299661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.299635 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq7m4\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-kube-api-access-hq7m4\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:03:53.299661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.299650 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:03:53.299661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.299664 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:03:53.300024 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.299679 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:03:53.300024 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.299692 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae724a0-57d2-4116-9537-31e7aa2aea87-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:03:53.408567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.408490 5035 generic.go:356] "Generic (PLEG): container finished" podID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerID="f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62" exitCode=0 Oct 30 02:03:53.408567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.408550 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" Oct 30 02:03:53.408792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.408574 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" event={"ID":"2ae724a0-57d2-4116-9537-31e7aa2aea87","Type":"ContainerDied","Data":"f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62"} Oct 30 02:03:53.408792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.408635 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h" event={"ID":"2ae724a0-57d2-4116-9537-31e7aa2aea87","Type":"ContainerDied","Data":"25bbdfea74500f84706dddd0cac89d8c9ac818a5c96a5a08f839dbd04b7830b5"} Oct 30 02:03:53.408792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.408662 5035 scope.go:122] "RemoveContainer" containerID="f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62" Oct 30 02:03:53.421392 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.421366 5035 scope.go:122] "RemoveContainer" containerID="b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9" Oct 30 02:03:53.433133 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.433109 5035 scope.go:122] "RemoveContainer" containerID="f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62" Oct 30 02:03:53.433492 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:03:53.433468 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62\": container with ID starting with f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62 not found: ID does not exist" containerID="f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62" Oct 30 02:03:53.433597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.433501 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62"} err="failed to get container status \"f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62\": rpc error: code = NotFound desc = could not find container \"f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62\": container with ID starting with f1c54a652fea8cadf9bf83daef26175c8e635735e363cacf6e666191765ecb62 not found: ID does not exist" Oct 30 02:03:53.433597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.433522 5035 scope.go:122] "RemoveContainer" containerID="b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9" Oct 30 02:03:53.433867 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:03:53.433840 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9\": container with ID starting with b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9 not found: ID does not exist" containerID="b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9" Oct 30 02:03:53.433924 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.433877 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9"} err="failed to get container status \"b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9\": rpc error: code = NotFound desc = could not find container \"b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9\": container with ID starting with b88c7ca009d5700e0eb8da718be65c4f112cac34386e239cfb9e6f4bdc32ebc9 not found: ID does not exist" Oct 30 02:03:53.493110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.493044 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h"] Oct 30 02:03:53.497267 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.497219 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-95daa2-openshift-default-656b88c449-9vb6h"] Oct 30 02:03:53.645402 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:53.645320 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" path="/var/lib/kubelet/pods/2ae724a0-57d2-4116-9537-31e7aa2aea87/volumes" Oct 30 02:03:55.917813 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.917777 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg"] Oct 30 02:03:55.918587 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.918559 5035 cpu_manager.go:414] "containerMap: removing container" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" Oct 30 02:03:55.918587 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.918580 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" Oct 30 02:03:55.918716 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.918598 5035 cpu_manager.go:414] "containerMap: removing container" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" Oct 30 02:03:55.918716 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.918604 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" Oct 30 02:03:55.918786 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.918723 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" Oct 30 02:03:55.918786 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.918735 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae724a0-57d2-4116-9537-31e7aa2aea87" containerName="istio-proxy" Oct 30 02:03:55.924019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.923995 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:03:55.943009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.942972 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg"] Oct 30 02:03:55.951430 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.951397 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3ad281-dockercfg-8krbc\"" Oct 30 02:03:55.951826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:55.951763 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3ad281\"" Oct 30 02:03:56.076554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.073979 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqjl\" (UniqueName: \"kubernetes.io/projected/a0e59e78-5498-4688-b4c7-e3a6911b6511-kube-api-access-5vqjl\") pod \"payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg\" (UID: \"a0e59e78-5498-4688-b4c7-e3a6911b6511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:03:56.076554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.074091 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59e78-5498-4688-b4c7-e3a6911b6511-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg\" (UID: \"a0e59e78-5498-4688-b4c7-e3a6911b6511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:03:56.205706 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.205182 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqjl\" (UniqueName: \"kubernetes.io/projected/a0e59e78-5498-4688-b4c7-e3a6911b6511-kube-api-access-5vqjl\") pod \"payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg\" (UID: \"a0e59e78-5498-4688-b4c7-e3a6911b6511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:03:56.206164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.206122 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59e78-5498-4688-b4c7-e3a6911b6511-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg\" (UID: \"a0e59e78-5498-4688-b4c7-e3a6911b6511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:03:56.207378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.207259 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59e78-5498-4688-b4c7-e3a6911b6511-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg\" (UID: \"a0e59e78-5498-4688-b4c7-e3a6911b6511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:03:56.224549 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.221678 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z"] Oct 30 02:03:56.230504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.228227 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:03:56.231257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.231201 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqjl\" (UniqueName: \"kubernetes.io/projected/a0e59e78-5498-4688-b4c7-e3a6911b6511-kube-api-access-5vqjl\") pod \"payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg\" (UID: \"a0e59e78-5498-4688-b4c7-e3a6911b6511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:03:56.233015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.232839 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z"] Oct 30 02:03:56.238895 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.238849 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:03:56.411934 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.411896 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z\" (UID: \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:03:56.412138 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.412011 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58g5\" (UniqueName: \"kubernetes.io/projected/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-kube-api-access-j58g5\") pod \"payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z\" (UID: \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:03:56.513298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.513191 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58g5\" (UniqueName: \"kubernetes.io/projected/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-kube-api-access-j58g5\") pod \"payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z\" (UID: \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:03:56.513468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.513390 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z\" (UID: \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:03:56.514206 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.514174 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z\" (UID: \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:03:56.538494 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.538465 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58g5\" (UniqueName: \"kubernetes.io/projected/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-kube-api-access-j58g5\") pod \"payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z\" (UID: \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:03:56.594519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.594468 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg"] Oct 30 02:03:56.598934 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:03:56.598894 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e59e78_5498_4688_b4c7_e3a6911b6511.slice/crio-caa4b11b81982b7a1cbdee7c6f63849f668336580a6ce7237c2bfc0699d0f828 WatchSource:0}: Error finding container caa4b11b81982b7a1cbdee7c6f63849f668336580a6ce7237c2bfc0699d0f828: Status 404 returned error can't find the container with id caa4b11b81982b7a1cbdee7c6f63849f668336580a6ce7237c2bfc0699d0f828 Oct 30 02:03:56.612197 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.612169 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:03:56.941187 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.941087 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5"] Oct 30 02:03:56.950398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.950350 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.000560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:56.995294 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0f3cd8-serving-cert\"" Oct 30 02:03:57.016935 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.016885 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5"] Oct 30 02:03:57.022944 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.022914 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z"] Oct 30 02:03:57.123201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.123161 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.123345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.123226 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.123446 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.123422 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-metrics-certs\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.123489 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.123471 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8g6\" (UniqueName: \"kubernetes.io/projected/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-kube-api-access-tb8g6\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.225644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.225604 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.225846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.225685 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.225846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.225804 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-metrics-certs\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.225985 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.225853 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8g6\" (UniqueName: \"kubernetes.io/projected/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-kube-api-access-tb8g6\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.227681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.227646 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.234536 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.233766 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.234536 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.234092 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-metrics-certs\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.253466 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.253434 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8g6\" (UniqueName: \"kubernetes.io/projected/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-kube-api-access-tb8g6\") pod \"maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.326041 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.325997 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:57.445370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.444921 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" event={"ID":"a0e59e78-5498-4688-b4c7-e3a6911b6511","Type":"ContainerStarted","Data":"34730face63504df59df485ecd98b8b58eeb57a7f41c70dab6a8861a12af884b"} Oct 30 02:03:57.445370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.445360 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:03:57.445653 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.445438 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" event={"ID":"a0e59e78-5498-4688-b4c7-e3a6911b6511","Type":"ContainerStarted","Data":"caa4b11b81982b7a1cbdee7c6f63849f668336580a6ce7237c2bfc0699d0f828"} Oct 30 02:03:57.446791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.446760 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" event={"ID":"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4","Type":"ContainerStarted","Data":"d686cb58960d347cf7cf7c6b4c6c3f09665aa493dea5c9ac4dd6fd2e51f35c8a"} Oct 30 02:03:57.446941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.446805 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" event={"ID":"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4","Type":"ContainerStarted","Data":"21943a5249ea20004742c922f58704322223c6893398ee4a935ce8f9703d06a5"} Oct 30 02:03:57.446941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.446826 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:03:57.496768 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.496651 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" podStartSLOduration=2.496615679 podStartE2EDuration="2.496615679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:57.492612893 +0000 UTC m=+2115.995545113" watchObservedRunningTime="2026-10-30 02:03:57.496615679 +0000 UTC m=+2115.999547901" Oct 30 02:03:57.539833 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.539771 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" podStartSLOduration=2.5397544659999998 podStartE2EDuration="2.539754466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:57.536832356 +0000 UTC m=+2116.039764577" watchObservedRunningTime="2026-10-30 02:03:57.539754466 +0000 UTC m=+2116.042686686" Oct 30 02:03:57.731162 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:57.715972 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5"] Oct 30 02:03:57.731162 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:03:57.730894 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9f5a20_1d2c_4b31_a37c_d049557b35d7.slice/crio-d741deb7b191e5030106f71e7fc2bf9b9620fd97c8700edb18eee9cd90186425 WatchSource:0}: Error finding container d741deb7b191e5030106f71e7fc2bf9b9620fd97c8700edb18eee9cd90186425: Status 404 returned error can't find the container with id d741deb7b191e5030106f71e7fc2bf9b9620fd97c8700edb18eee9cd90186425 Oct 30 02:03:58.455796 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:58.455753 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" event={"ID":"1a9f5a20-1d2c-4b31-a37c-d049557b35d7","Type":"ContainerStarted","Data":"010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc"} Oct 30 02:03:58.456323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:58.455816 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" event={"ID":"1a9f5a20-1d2c-4b31-a37c-d049557b35d7","Type":"ContainerStarted","Data":"d741deb7b191e5030106f71e7fc2bf9b9620fd97c8700edb18eee9cd90186425"} Oct 30 02:03:58.507475 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:58.507410 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" podStartSLOduration=2.116876924 podStartE2EDuration="2.507392364s" totalImagesPullingTime="390.51544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:58.503657991 +0000 UTC m=+2117.006590210" watchObservedRunningTime="2026-10-30 02:03:58.507392364 +0000 UTC m=+2117.010324584" Oct 30 02:03:59.463969 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:59.463933 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:03:59.739737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:03:59.739646 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb"] Oct 30 02:04:02.823353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:02.823266 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:04:02.823353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:02.823324 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 02:04:03.532585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:03.532507 5035 generic.go:356] "Generic (PLEG): container finished" podID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerID="a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0" exitCode=0 Oct 30 02:04:03.532780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:03.532583 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" event={"ID":"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c","Type":"ContainerDied","Data":"a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0"} Oct 30 02:04:03.533128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:03.533113 5035 scope.go:122] "RemoveContainer" containerID="a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0" Oct 30 02:04:03.534915 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:03.534890 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:04:03.534998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:03.534962 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:04:03.535037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:03.534997 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:04:04.542785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:04.542741 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" event={"ID":"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c","Type":"ContainerStarted","Data":"578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc"} Oct 30 02:04:04.740920 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:04.740865 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" podUID="0ee74cc4-a5ee-4104-988a-9301ab5d7548" containerName="payload-pre-processing" containerID="cri-o://8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03" gracePeriod=25 Oct 30 02:04:05.106271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.106025 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:04:05.207013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.206980 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee74cc4-a5ee-4104-988a-9301ab5d7548-kube-api-access-7t6gq\" (UniqueName: \"kubernetes.io/projected/0ee74cc4-a5ee-4104-988a-9301ab5d7548-kube-api-access-7t6gq\") pod \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\" (UID: \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\") " Oct 30 02:04:05.207239 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.207066 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee74cc4-a5ee-4104-988a-9301ab5d7548-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee74cc4-a5ee-4104-988a-9301ab5d7548-plugins-config-volume\") pod \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\" (UID: \"0ee74cc4-a5ee-4104-988a-9301ab5d7548\") " Oct 30 02:04:05.207515 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.207487 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee74cc4-a5ee-4104-988a-9301ab5d7548-plugins-config-volume" pod "0ee74cc4-a5ee-4104-988a-9301ab5d7548" (UID: "0ee74cc4-a5ee-4104-988a-9301ab5d7548"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:05.209315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.209288 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee74cc4-a5ee-4104-988a-9301ab5d7548-kube-api-access-7t6gq" pod "0ee74cc4-a5ee-4104-988a-9301ab5d7548" (UID: "0ee74cc4-a5ee-4104-988a-9301ab5d7548"). InnerVolumeSpecName "kube-api-access-7t6gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:05.308720 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.308680 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t6gq\" (UniqueName: \"kubernetes.io/projected/0ee74cc4-a5ee-4104-988a-9301ab5d7548-kube-api-access-7t6gq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:05.308720 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.308713 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ee74cc4-a5ee-4104-988a-9301ab5d7548-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:05.508922 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.508838 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:04:05.537235 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.537196 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:04:05.537626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.537602 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:04:05.537681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.537651 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 02:04:05.552617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.552575 5035 generic.go:356] "Generic (PLEG): container finished" podID="0ee74cc4-a5ee-4104-988a-9301ab5d7548" containerID="8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03" exitCode=0 Oct 30 02:04:05.553026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.552651 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" Oct 30 02:04:05.553026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.552657 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" event={"ID":"0ee74cc4-a5ee-4104-988a-9301ab5d7548","Type":"ContainerDied","Data":"8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03"} Oct 30 02:04:05.553026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.552717 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb" event={"ID":"0ee74cc4-a5ee-4104-988a-9301ab5d7548","Type":"ContainerDied","Data":"52b0e529bb5b4242f66243750c8072b7e691d16eaf422dc87a8a44f7bd3e1a12"} Oct 30 02:04:05.553026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.552740 5035 scope.go:122] "RemoveContainer" containerID="8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03" Oct 30 02:04:05.613788 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.613758 5035 scope.go:122] "RemoveContainer" containerID="8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03" Oct 30 02:04:05.614124 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:05.614099 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03\": container with ID starting with 8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03 not found: ID does not exist" containerID="8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03" Oct 30 02:04:05.614209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.614133 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03"} err="failed to get container status \"8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03\": rpc error: code = NotFound desc = could not find container \"8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03\": container with ID starting with 8ea9de3a4d67fc7b6617282f6ae9aadb3f7c3b95e44736939d59a261b09e9d03 not found: ID does not exist" Oct 30 02:04:05.647259 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.647221 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb"] Oct 30 02:04:05.649036 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:05.649000 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea4bc0-7d5858859-mz5mb"] Oct 30 02:04:06.537080 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:06.537036 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:04:06.537285 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:06.537101 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 02:04:07.537841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:07.537799 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:04:07.538285 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:07.537857 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 02:04:07.644947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:07.644908 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee74cc4-a5ee-4104-988a-9301ab5d7548" path="/var/lib/kubelet/pods/0ee74cc4-a5ee-4104-988a-9301ab5d7548/volumes" Oct 30 02:04:08.459566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:08.459513 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:04:08.459771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:08.459622 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:04:08.537507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:08.537465 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:04:08.537763 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:08.537545 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 02:04:09.537770 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:09.537715 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 02:04:09.538263 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:09.537777 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 02:04:10.541735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:10.541703 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:04:10.542228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:10.542193 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:04:10.543381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:10.543357 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:04:14.507056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.507018 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q"] Oct 30 02:04:14.507617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.507330 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" podUID="af003945-c61f-4abe-b793-b3538cfc6d52" containerName="maas-api" containerID="cri-o://2485b4cdd9ee5d813956e667844d249a66238b017b240fec9e88c3faa36f15c0" gracePeriod=30 Oct 30 02:04:14.655563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.655497 5035 generic.go:356] "Generic (PLEG): container finished" podID="af003945-c61f-4abe-b793-b3538cfc6d52" containerID="2485b4cdd9ee5d813956e667844d249a66238b017b240fec9e88c3faa36f15c0" exitCode=0 Oct 30 02:04:14.655762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.655566 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" event={"ID":"af003945-c61f-4abe-b793-b3538cfc6d52","Type":"ContainerDied","Data":"2485b4cdd9ee5d813956e667844d249a66238b017b240fec9e88c3faa36f15c0"} Oct 30 02:04:14.803972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.803940 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:04:14.899188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.898772 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-metrics-certs\") pod \"af003945-c61f-4abe-b793-b3538cfc6d52\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " Oct 30 02:04:14.899188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.898901 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af003945-c61f-4abe-b793-b3538cfc6d52-kube-api-access-thq2c\" (UniqueName: \"kubernetes.io/projected/af003945-c61f-4abe-b793-b3538cfc6d52-kube-api-access-thq2c\") pod \"af003945-c61f-4abe-b793-b3538cfc6d52\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " Oct 30 02:04:14.899188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.898956 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af003945-c61f-4abe-b793-b3538cfc6d52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af003945-c61f-4abe-b793-b3538cfc6d52-openshift-service-ca\") pod \"af003945-c61f-4abe-b793-b3538cfc6d52\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " Oct 30 02:04:14.899188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.898999 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-maas-api-tls\") pod \"af003945-c61f-4abe-b793-b3538cfc6d52\" (UID: \"af003945-c61f-4abe-b793-b3538cfc6d52\") " Oct 30 02:04:14.901503 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.901465 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af003945-c61f-4abe-b793-b3538cfc6d52-openshift-service-ca" pod "af003945-c61f-4abe-b793-b3538cfc6d52" (UID: "af003945-c61f-4abe-b793-b3538cfc6d52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:14.904599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.904490 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-maas-api-tls" pod "af003945-c61f-4abe-b793-b3538cfc6d52" (UID: "af003945-c61f-4abe-b793-b3538cfc6d52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:14.931350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.931283 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-metrics-certs" pod "af003945-c61f-4abe-b793-b3538cfc6d52" (UID: "af003945-c61f-4abe-b793-b3538cfc6d52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:14.942620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:14.942553 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af003945-c61f-4abe-b793-b3538cfc6d52-kube-api-access-thq2c" pod "af003945-c61f-4abe-b793-b3538cfc6d52" (UID: "af003945-c61f-4abe-b793-b3538cfc6d52"). InnerVolumeSpecName "kube-api-access-thq2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:15.011610 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:15.001660 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af003945-c61f-4abe-b793-b3538cfc6d52-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:15.011610 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:15.001702 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:15.011610 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:15.001716 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af003945-c61f-4abe-b793-b3538cfc6d52-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:15.011610 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:15.001732 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thq2c\" (UniqueName: \"kubernetes.io/projected/af003945-c61f-4abe-b793-b3538cfc6d52-kube-api-access-thq2c\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:15.667132 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:15.667100 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" Oct 30 02:04:15.668814 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:15.667155 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q" event={"ID":"af003945-c61f-4abe-b793-b3538cfc6d52","Type":"ContainerDied","Data":"9f273d4ed9057e149e6f04a53ef208e6ec8ea08d2ddd1e5e5dc5f6865c68fb9c"} Oct 30 02:04:15.668814 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:15.667241 5035 scope.go:122] "RemoveContainer" containerID="2485b4cdd9ee5d813956e667844d249a66238b017b240fec9e88c3faa36f15c0" Oct 30 02:04:15.728255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:15.728211 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q"] Oct 30 02:04:15.740908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:15.740871 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea4bc0-84bdb84fc9-7j64q"] Oct 30 02:04:16.958486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.958451 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5"] Oct 30 02:04:16.959203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.959186 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0ee74cc4-a5ee-4104-988a-9301ab5d7548" containerName="payload-pre-processing" Oct 30 02:04:16.959268 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.959204 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee74cc4-a5ee-4104-988a-9301ab5d7548" containerName="payload-pre-processing" Oct 30 02:04:16.959268 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.959233 5035 cpu_manager.go:414] "containerMap: removing container" podUID="af003945-c61f-4abe-b793-b3538cfc6d52" containerName="maas-api" Oct 30 02:04:16.959268 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.959240 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af003945-c61f-4abe-b793-b3538cfc6d52" containerName="maas-api" Oct 30 02:04:16.959414 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.959351 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="af003945-c61f-4abe-b793-b3538cfc6d52" containerName="maas-api" Oct 30 02:04:16.959414 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.959367 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee74cc4-a5ee-4104-988a-9301ab5d7548" containerName="payload-pre-processing" Oct 30 02:04:16.966439 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.966410 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:16.990438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.990224 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5"] Oct 30 02:04:16.995122 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:16.995065 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3ad281-serving-cert\"" Oct 30 02:04:17.049798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.049755 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-metrics-certs\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.049978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.049895 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.050044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.050026 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmlw6\" (UniqueName: \"kubernetes.io/projected/7e8d8d03-2379-40d3-a54b-82a119a35c9a-kube-api-access-wmlw6\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.050106 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.050093 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8d03-2379-40d3-a54b-82a119a35c9a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.154652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.151249 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-metrics-certs\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.154652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.151462 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.154652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.151631 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmlw6\" (UniqueName: \"kubernetes.io/projected/7e8d8d03-2379-40d3-a54b-82a119a35c9a-kube-api-access-wmlw6\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.154652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.151729 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8d03-2379-40d3-a54b-82a119a35c9a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.154652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.152583 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8d03-2379-40d3-a54b-82a119a35c9a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.171733 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.169207 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.182633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.180322 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-metrics-certs\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.185959 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.185924 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmlw6\" (UniqueName: \"kubernetes.io/projected/7e8d8d03-2379-40d3-a54b-82a119a35c9a-kube-api-access-wmlw6\") pod \"maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.293809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.293693 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:17.616853 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.616821 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5"] Oct 30 02:04:17.618717 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:04:17.618682 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8d8d03_2379_40d3_a54b_82a119a35c9a.slice/crio-75d5d167eeb9b2110f9dce64fd9c150357d251d00bda162ae91f03bdc4b32860 WatchSource:0}: Error finding container 75d5d167eeb9b2110f9dce64fd9c150357d251d00bda162ae91f03bdc4b32860: Status 404 returned error can't find the container with id 75d5d167eeb9b2110f9dce64fd9c150357d251d00bda162ae91f03bdc4b32860 Oct 30 02:04:17.632696 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:17.632658 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:04:17.644117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.644085 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af003945-c61f-4abe-b793-b3538cfc6d52" path="/var/lib/kubelet/pods/af003945-c61f-4abe-b793-b3538cfc6d52/volumes" Oct 30 02:04:17.684599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:17.684553 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" event={"ID":"7e8d8d03-2379-40d3-a54b-82a119a35c9a","Type":"ContainerStarted","Data":"75d5d167eeb9b2110f9dce64fd9c150357d251d00bda162ae91f03bdc4b32860"} Oct 30 02:04:18.703382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:18.703345 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" event={"ID":"7e8d8d03-2379-40d3-a54b-82a119a35c9a","Type":"ContainerStarted","Data":"67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940"} Oct 30 02:04:18.703834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:18.703440 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:18.754625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:18.754518 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" podStartSLOduration=2.40940087 podStartE2EDuration="2.754494951s" totalImagesPullingTime="345.094081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:18.752164129 +0000 UTC m=+2137.255096350" watchObservedRunningTime="2026-10-30 02:04:18.754494951 +0000 UTC m=+2137.257427171" Oct 30 02:04:19.659610 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:19.659571 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k"] Oct 30 02:04:24.113314 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:24.113274 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f"] Oct 30 02:04:24.113863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:24.113570 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" containerID="cri-o://2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3" gracePeriod=30 Oct 30 02:04:24.660389 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:24.660337 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" podUID="47c22c39-4909-447a-a55b-c20294cdcd1f" containerName="payload-processing" containerID="cri-o://d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136" gracePeriod=25 Oct 30 02:04:24.718476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:24.718444 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:04:24.980911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:24.980881 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:04:25.069374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.040135 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c22c39-4909-447a-a55b-c20294cdcd1f-kube-api-access-swnk5\" (UniqueName: \"kubernetes.io/projected/47c22c39-4909-447a-a55b-c20294cdcd1f-kube-api-access-swnk5\") pod \"47c22c39-4909-447a-a55b-c20294cdcd1f\" (UID: \"47c22c39-4909-447a-a55b-c20294cdcd1f\") " Oct 30 02:04:25.069374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.040385 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c22c39-4909-447a-a55b-c20294cdcd1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c22c39-4909-447a-a55b-c20294cdcd1f-plugins-config-volume\") pod \"47c22c39-4909-447a-a55b-c20294cdcd1f\" (UID: \"47c22c39-4909-447a-a55b-c20294cdcd1f\") " Oct 30 02:04:25.069374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.041139 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c22c39-4909-447a-a55b-c20294cdcd1f-plugins-config-volume" pod "47c22c39-4909-447a-a55b-c20294cdcd1f" (UID: "47c22c39-4909-447a-a55b-c20294cdcd1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:25.069374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.043807 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c22c39-4909-447a-a55b-c20294cdcd1f-kube-api-access-swnk5" pod "47c22c39-4909-447a-a55b-c20294cdcd1f" (UID: "47c22c39-4909-447a-a55b-c20294cdcd1f"). InnerVolumeSpecName "kube-api-access-swnk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:25.143729 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.143689 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swnk5\" (UniqueName: \"kubernetes.io/projected/47c22c39-4909-447a-a55b-c20294cdcd1f-kube-api-access-swnk5\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:25.143729 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.143731 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c22c39-4909-447a-a55b-c20294cdcd1f-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:25.768032 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.767992 5035 generic.go:356] "Generic (PLEG): container finished" podID="47c22c39-4909-447a-a55b-c20294cdcd1f" containerID="d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136" exitCode=0 Oct 30 02:04:25.768251 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.768043 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" Oct 30 02:04:25.768251 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.768075 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" event={"ID":"47c22c39-4909-447a-a55b-c20294cdcd1f","Type":"ContainerDied","Data":"d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136"} Oct 30 02:04:25.768251 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.768140 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k" event={"ID":"47c22c39-4909-447a-a55b-c20294cdcd1f","Type":"ContainerDied","Data":"705c806c7a96bf59c8c45abec1c04eb1fdfe6717d9e4129e986f977e0e963754"} Oct 30 02:04:25.768251 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.768170 5035 scope.go:122] "RemoveContainer" containerID="d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136" Oct 30 02:04:25.787682 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.785872 5035 scope.go:122] "RemoveContainer" containerID="d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136" Oct 30 02:04:25.787682 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:25.786465 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136\": container with ID starting with d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136 not found: ID does not exist" containerID="d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136" Oct 30 02:04:25.787682 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.786509 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136"} err="failed to get container status \"d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136\": rpc error: code = NotFound desc = could not find container \"d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136\": container with ID starting with d3524a72af0fceefe08db8e1456b69cda4e802bb726e875b885a295631704136 not found: ID does not exist" Oct 30 02:04:25.836087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.836043 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k"] Oct 30 02:04:25.849143 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:25.849092 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70a069-588bb4c774-k8f7k"] Oct 30 02:04:27.644313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:27.644269 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c22c39-4909-447a-a55b-c20294cdcd1f" path="/var/lib/kubelet/pods/47c22c39-4909-447a-a55b-c20294cdcd1f/volumes" Oct 30 02:04:28.445565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:28.445511 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c"] Oct 30 02:04:29.483965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.483935 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:04:29.620869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.620829 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-envoy\") pod \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " Oct 30 02:04:29.621071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.620927 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-certs\") pod \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " Oct 30 02:04:29.621071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621019 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-data\") pod \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " Oct 30 02:04:29.621071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621054 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-podinfo\") pod \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " Oct 30 02:04:29.621200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621107 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istiod-ca-cert\") pod \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " Oct 30 02:04:29.621200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621152 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-credential-socket\") pod \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " Oct 30 02:04:29.621311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621220 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-token\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-token\") pod \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " Oct 30 02:04:29.621311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621261 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-socket\") pod \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " Oct 30 02:04:29.621311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621256 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-certs" pod "f5fb784e-0d01-4def-bdc1-63259c9e1fbf" (UID: "f5fb784e-0d01-4def-bdc1-63259c9e1fbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:29.621482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621365 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-data" pod "f5fb784e-0d01-4def-bdc1-63259c9e1fbf" (UID: "f5fb784e-0d01-4def-bdc1-63259c9e1fbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:29.621482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621447 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-kube-api-access-46kkc\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-kube-api-access-46kkc\") pod \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\" (UID: \"f5fb784e-0d01-4def-bdc1-63259c9e1fbf\") " Oct 30 02:04:29.621661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621593 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-socket" pod "f5fb784e-0d01-4def-bdc1-63259c9e1fbf" (UID: "f5fb784e-0d01-4def-bdc1-63259c9e1fbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:29.621661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621614 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istiod-ca-cert" pod "f5fb784e-0d01-4def-bdc1-63259c9e1fbf" (UID: "f5fb784e-0d01-4def-bdc1-63259c9e1fbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:29.621921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.621890 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-credential-socket" pod "f5fb784e-0d01-4def-bdc1-63259c9e1fbf" (UID: "f5fb784e-0d01-4def-bdc1-63259c9e1fbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:29.622236 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.622212 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:29.622339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.622241 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:29.622339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.622255 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:29.622339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.622268 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:29.622339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.622283 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:29.623609 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.623568 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-envoy" pod "f5fb784e-0d01-4def-bdc1-63259c9e1fbf" (UID: "f5fb784e-0d01-4def-bdc1-63259c9e1fbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:29.623720 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.623613 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-token" pod "f5fb784e-0d01-4def-bdc1-63259c9e1fbf" (UID: "f5fb784e-0d01-4def-bdc1-63259c9e1fbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:29.623823 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.623797 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-podinfo" pod "f5fb784e-0d01-4def-bdc1-63259c9e1fbf" (UID: "f5fb784e-0d01-4def-bdc1-63259c9e1fbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:29.623877 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.623820 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-kube-api-access-46kkc" pod "f5fb784e-0d01-4def-bdc1-63259c9e1fbf" (UID: "f5fb784e-0d01-4def-bdc1-63259c9e1fbf"). InnerVolumeSpecName "kube-api-access-46kkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:29.724146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.724058 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:29.724146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.724096 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:29.724146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.724112 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:29.724146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.724127 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46kkc\" (UniqueName: \"kubernetes.io/projected/f5fb784e-0d01-4def-bdc1-63259c9e1fbf-kube-api-access-46kkc\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:29.830581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.818476 5035 generic.go:356] "Generic (PLEG): container finished" podID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerID="2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3" exitCode=0 Oct 30 02:04:29.830581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.818674 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" event={"ID":"f5fb784e-0d01-4def-bdc1-63259c9e1fbf","Type":"ContainerDied","Data":"2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3"} Oct 30 02:04:29.830581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.818732 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" event={"ID":"f5fb784e-0d01-4def-bdc1-63259c9e1fbf","Type":"ContainerDied","Data":"6445359c566fafc210cc379d669bab95c7903056feefca4025ba4701af4cc145"} Oct 30 02:04:29.830581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.818767 5035 scope.go:122] "RemoveContainer" containerID="2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3" Oct 30 02:04:29.830581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.818942 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f" Oct 30 02:04:29.872491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.872458 5035 scope.go:122] "RemoveContainer" containerID="348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84" Oct 30 02:04:29.890854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.890827 5035 scope.go:122] "RemoveContainer" containerID="2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3" Oct 30 02:04:29.891238 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:29.891213 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3\": container with ID starting with 2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3 not found: ID does not exist" containerID="2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3" Oct 30 02:04:29.891352 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.891245 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3"} err="failed to get container status \"2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3\": rpc error: code = NotFound desc = could not find container \"2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3\": container with ID starting with 2c9d50e7cca074fd59633d60b2c448cfecaaa64893ddc5a6fc41140599f5d8d3 not found: ID does not exist" Oct 30 02:04:29.891352 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.891269 5035 scope.go:122] "RemoveContainer" containerID="348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84" Oct 30 02:04:29.891636 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:29.891610 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84\": container with ID starting with 348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84 not found: ID does not exist" containerID="348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84" Oct 30 02:04:29.891761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.891638 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84"} err="failed to get container status \"348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84\": rpc error: code = NotFound desc = could not find container \"348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84\": container with ID starting with 348c707face3c38a7197cf80a8e6768b4852f43994ad553c1b30efe507eb6c84 not found: ID does not exist" Oct 30 02:04:29.918083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.918050 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f"] Oct 30 02:04:29.934752 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:29.934711 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea4bc0-openshift-default-7db8b979fc-6f88f"] Oct 30 02:04:31.645325 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:31.645287 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" path="/var/lib/kubelet/pods/f5fb784e-0d01-4def-bdc1-63259c9e1fbf/volumes" Oct 30 02:04:33.446828 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.446776 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" podUID="c58096a8-ee94-4650-b166-ff870a6b5f08" containerName="payload-pre-processing" containerID="cri-o://09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988" gracePeriod=25 Oct 30 02:04:33.770625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.770588 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:04:33.854067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.853863 5035 generic.go:356] "Generic (PLEG): container finished" podID="c58096a8-ee94-4650-b166-ff870a6b5f08" containerID="09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988" exitCode=0 Oct 30 02:04:33.854279 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.854133 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" event={"ID":"c58096a8-ee94-4650-b166-ff870a6b5f08","Type":"ContainerDied","Data":"09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988"} Oct 30 02:04:33.854279 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.854184 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" event={"ID":"c58096a8-ee94-4650-b166-ff870a6b5f08","Type":"ContainerDied","Data":"365dc008bc02617d9f85211e16d2ae27ba9a691f91bca685eee223bb101617fa"} Oct 30 02:04:33.854279 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.854209 5035 scope.go:122] "RemoveContainer" containerID="09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988" Oct 30 02:04:33.854485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.854346 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c" Oct 30 02:04:33.868694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.868665 5035 scope.go:122] "RemoveContainer" containerID="09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988" Oct 30 02:04:33.869190 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:33.869158 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988\": container with ID starting with 09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988 not found: ID does not exist" containerID="09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988" Oct 30 02:04:33.869321 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.869203 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988"} err="failed to get container status \"09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988\": rpc error: code = NotFound desc = could not find container \"09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988\": container with ID starting with 09d1aaf03a9fd6e67a7c7b4e590f9883b0df479de03e934a31638cb66e3be988 not found: ID does not exist" Oct 30 02:04:33.893164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.893128 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58096a8-ee94-4650-b166-ff870a6b5f08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58096a8-ee94-4650-b166-ff870a6b5f08-plugins-config-volume\") pod \"c58096a8-ee94-4650-b166-ff870a6b5f08\" (UID: \"c58096a8-ee94-4650-b166-ff870a6b5f08\") " Oct 30 02:04:33.893359 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.893280 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58096a8-ee94-4650-b166-ff870a6b5f08-kube-api-access-5mk2z\" (UniqueName: \"kubernetes.io/projected/c58096a8-ee94-4650-b166-ff870a6b5f08-kube-api-access-5mk2z\") pod \"c58096a8-ee94-4650-b166-ff870a6b5f08\" (UID: \"c58096a8-ee94-4650-b166-ff870a6b5f08\") " Oct 30 02:04:33.893641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.893613 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58096a8-ee94-4650-b166-ff870a6b5f08-plugins-config-volume" pod "c58096a8-ee94-4650-b166-ff870a6b5f08" (UID: "c58096a8-ee94-4650-b166-ff870a6b5f08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:33.893866 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.893850 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58096a8-ee94-4650-b166-ff870a6b5f08-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:33.895508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.895475 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58096a8-ee94-4650-b166-ff870a6b5f08-kube-api-access-5mk2z" pod "c58096a8-ee94-4650-b166-ff870a6b5f08" (UID: "c58096a8-ee94-4650-b166-ff870a6b5f08"). InnerVolumeSpecName "kube-api-access-5mk2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:33.995254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:33.995160 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mk2z\" (UniqueName: \"kubernetes.io/projected/c58096a8-ee94-4650-b166-ff870a6b5f08-kube-api-access-5mk2z\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:34.212476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:34.212435 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c"] Oct 30 02:04:34.226121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:34.226081 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0f3cd8-788765798c-kbs5c"] Oct 30 02:04:35.645349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:35.645309 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58096a8-ee94-4650-b166-ff870a6b5f08" path="/var/lib/kubelet/pods/c58096a8-ee94-4650-b166-ff870a6b5f08/volumes" Oct 30 02:04:43.017877 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:43.017833 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz"] Oct 30 02:04:43.018398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:43.018054 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" containerID="cri-o://1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7" gracePeriod=30 Oct 30 02:04:46.270744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:46.270668 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl"] Oct 30 02:04:46.272491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:46.272446 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" containerID="cri-o://f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0" gracePeriod=30 Oct 30 02:04:47.630920 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:47.630867 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:04:48.314105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.314072 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 02:04:48.429752 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.429702 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-data\") pod \"24815994-57e7-4a2a-9a36-0be361ef7ba1\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " Oct 30 02:04:48.429978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.429821 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-kube-api-access-kx97n\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-kube-api-access-kx97n\") pod \"24815994-57e7-4a2a-9a36-0be361ef7ba1\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " Oct 30 02:04:48.429978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.429905 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-token\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-token\") pod \"24815994-57e7-4a2a-9a36-0be361ef7ba1\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " Oct 30 02:04:48.429978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.429978 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24815994-57e7-4a2a-9a36-0be361ef7ba1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24815994-57e7-4a2a-9a36-0be361ef7ba1-istiod-ca-cert\") pod \"24815994-57e7-4a2a-9a36-0be361ef7ba1\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " Oct 30 02:04:48.430280 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.430027 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-podinfo\") pod \"24815994-57e7-4a2a-9a36-0be361ef7ba1\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " Oct 30 02:04:48.430280 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.430027 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-data" pod "24815994-57e7-4a2a-9a36-0be361ef7ba1" (UID: "24815994-57e7-4a2a-9a36-0be361ef7ba1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.430280 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.430109 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-credential-socket\") pod \"24815994-57e7-4a2a-9a36-0be361ef7ba1\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " Oct 30 02:04:48.430280 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.430146 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-socket\") pod \"24815994-57e7-4a2a-9a36-0be361ef7ba1\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " Oct 30 02:04:48.430280 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.430179 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-envoy\") pod \"24815994-57e7-4a2a-9a36-0be361ef7ba1\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " Oct 30 02:04:48.430280 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.430255 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-certs\") pod \"24815994-57e7-4a2a-9a36-0be361ef7ba1\" (UID: \"24815994-57e7-4a2a-9a36-0be361ef7ba1\") " Oct 30 02:04:48.430786 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.430653 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-credential-socket" pod "24815994-57e7-4a2a-9a36-0be361ef7ba1" (UID: "24815994-57e7-4a2a-9a36-0be361ef7ba1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.430980 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.430943 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24815994-57e7-4a2a-9a36-0be361ef7ba1-istiod-ca-cert" pod "24815994-57e7-4a2a-9a36-0be361ef7ba1" (UID: "24815994-57e7-4a2a-9a36-0be361ef7ba1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:48.431082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.431025 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.431082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.431048 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.431313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.431291 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-socket" pod "24815994-57e7-4a2a-9a36-0be361ef7ba1" (UID: "24815994-57e7-4a2a-9a36-0be361ef7ba1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.431581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.431558 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-certs" pod "24815994-57e7-4a2a-9a36-0be361ef7ba1" (UID: "24815994-57e7-4a2a-9a36-0be361ef7ba1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.434659 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.434625 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-kube-api-access-kx97n" pod "24815994-57e7-4a2a-9a36-0be361ef7ba1" (UID: "24815994-57e7-4a2a-9a36-0be361ef7ba1"). InnerVolumeSpecName "kube-api-access-kx97n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:48.436620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.436588 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-podinfo" pod "24815994-57e7-4a2a-9a36-0be361ef7ba1" (UID: "24815994-57e7-4a2a-9a36-0be361ef7ba1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:48.436756 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.436676 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-token" pod "24815994-57e7-4a2a-9a36-0be361ef7ba1" (UID: "24815994-57e7-4a2a-9a36-0be361ef7ba1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:48.438005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.437969 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-envoy" pod "24815994-57e7-4a2a-9a36-0be361ef7ba1" (UID: "24815994-57e7-4a2a-9a36-0be361ef7ba1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.544667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.532981 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.544667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.533025 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx97n\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-kube-api-access-kx97n\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.544667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.533044 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.544667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.533058 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24815994-57e7-4a2a-9a36-0be361ef7ba1-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.544667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.533074 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.544667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.533089 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.544667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.533104 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24815994-57e7-4a2a-9a36-0be361ef7ba1-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.983180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.983139 5035 generic.go:356] "Generic (PLEG): container finished" podID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerID="1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7" exitCode=0 Oct 30 02:04:48.983705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.983224 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" Oct 30 02:04:48.983705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.983225 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" event={"ID":"24815994-57e7-4a2a-9a36-0be361ef7ba1","Type":"ContainerDied","Data":"1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7"} Oct 30 02:04:48.983705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.983363 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz" event={"ID":"24815994-57e7-4a2a-9a36-0be361ef7ba1","Type":"ContainerDied","Data":"04d1d4263786719a96e89eb289492292c60ed0fec0dbe37b4e583b188442ee87"} Oct 30 02:04:48.983705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:48.983394 5035 scope.go:122] "RemoveContainer" containerID="1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7" Oct 30 02:04:49.001504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:49.001484 5035 scope.go:122] "RemoveContainer" containerID="1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7" Oct 30 02:04:49.001881 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:49.001850 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7\": container with ID starting with 1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7 not found: ID does not exist" containerID="1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7" Oct 30 02:04:49.001993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:49.001888 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7"} err="failed to get container status \"1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7\": rpc error: code = NotFound desc = could not find container \"1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7\": container with ID starting with 1d32c49f1031aa1df366b196fd564fed007179f1530a164d28962846ec9472d7 not found: ID does not exist" Oct 30 02:04:49.057134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:49.057095 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz"] Oct 30 02:04:49.070838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:49.070794 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wd2rz"] Oct 30 02:04:49.644694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:49.644660 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" path="/var/lib/kubelet/pods/24815994-57e7-4a2a-9a36-0be361ef7ba1/volumes" Oct 30 02:04:51.560844 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.560771 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:04:51.677780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.677740 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-token\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-token\") pod \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " Oct 30 02:04:51.678110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678089 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-certs\") pod \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " Oct 30 02:04:51.678242 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678158 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-credential-socket\") pod \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " Oct 30 02:04:51.678297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678239 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-data\") pod \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " Oct 30 02:04:51.678297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678285 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-kube-api-access-7zmgm\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-kube-api-access-7zmgm\") pod \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " Oct 30 02:04:51.678398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678380 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istiod-ca-cert\") pod \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " Oct 30 02:04:51.678457 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678426 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-socket\") pod \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " Oct 30 02:04:51.678457 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678443 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-certs" pod "1b81a0ab-3b26-403b-bfb3-b36cf060240f" (UID: "1b81a0ab-3b26-403b-bfb3-b36cf060240f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.678611 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678478 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-envoy\") pod \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " Oct 30 02:04:51.678611 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678489 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-credential-socket" pod "1b81a0ab-3b26-403b-bfb3-b36cf060240f" (UID: "1b81a0ab-3b26-403b-bfb3-b36cf060240f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.678611 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678539 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-podinfo\") pod \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\" (UID: \"1b81a0ab-3b26-403b-bfb3-b36cf060240f\") " Oct 30 02:04:51.678780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678612 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-data" pod "1b81a0ab-3b26-403b-bfb3-b36cf060240f" (UID: "1b81a0ab-3b26-403b-bfb3-b36cf060240f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.678780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678686 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-socket" pod "1b81a0ab-3b26-403b-bfb3-b36cf060240f" (UID: "1b81a0ab-3b26-403b-bfb3-b36cf060240f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.678899 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.678815 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istiod-ca-cert" pod "1b81a0ab-3b26-403b-bfb3-b36cf060240f" (UID: "1b81a0ab-3b26-403b-bfb3-b36cf060240f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:51.679318 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.679295 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.679386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.679327 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.679386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.679342 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.679386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.679356 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.679386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.679369 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.680304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.680284 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-token" pod "1b81a0ab-3b26-403b-bfb3-b36cf060240f" (UID: "1b81a0ab-3b26-403b-bfb3-b36cf060240f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:51.680759 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.680728 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-podinfo" pod "1b81a0ab-3b26-403b-bfb3-b36cf060240f" (UID: "1b81a0ab-3b26-403b-bfb3-b36cf060240f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:51.681003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.680984 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-envoy" pod "1b81a0ab-3b26-403b-bfb3-b36cf060240f" (UID: "1b81a0ab-3b26-403b-bfb3-b36cf060240f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.681056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.681019 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-kube-api-access-7zmgm" pod "1b81a0ab-3b26-403b-bfb3-b36cf060240f" (UID: "1b81a0ab-3b26-403b-bfb3-b36cf060240f"). InnerVolumeSpecName "kube-api-access-7zmgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:51.781225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.781182 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmgm\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-kube-api-access-7zmgm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.781225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.781220 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.781225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.781229 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.781497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:51.781241 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b81a0ab-3b26-403b-bfb3-b36cf060240f-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:52.020874 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.020835 5035 generic.go:356] "Generic (PLEG): container finished" podID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerID="f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0" exitCode=0 Oct 30 02:04:52.021099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.020931 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" Oct 30 02:04:52.021099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.020945 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" event={"ID":"1b81a0ab-3b26-403b-bfb3-b36cf060240f","Type":"ContainerDied","Data":"f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0"} Oct 30 02:04:52.021099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.021087 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl" event={"ID":"1b81a0ab-3b26-403b-bfb3-b36cf060240f","Type":"ContainerDied","Data":"7e990cc6c1f962e0532fdfee621c5d5946047b4581d1d28acfa48d58a2f6da50"} Oct 30 02:04:52.021228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.021129 5035 scope.go:122] "RemoveContainer" containerID="f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0" Oct 30 02:04:52.036485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.036460 5035 scope.go:122] "RemoveContainer" containerID="965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67" Oct 30 02:04:52.047495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.047472 5035 scope.go:122] "RemoveContainer" containerID="f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0" Oct 30 02:04:52.047908 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:52.047884 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0\": container with ID starting with f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0 not found: ID does not exist" containerID="f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0" Oct 30 02:04:52.047971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.047922 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0"} err="failed to get container status \"f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0\": rpc error: code = NotFound desc = could not find container \"f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0\": container with ID starting with f3c43b8056c9a1ce2af7cf267d7e04b74afa7b797e46c7df2f271adb8cc2abe0 not found: ID does not exist" Oct 30 02:04:52.047971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.047943 5035 scope.go:122] "RemoveContainer" containerID="965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67" Oct 30 02:04:52.048250 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:52.048230 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67\": container with ID starting with 965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67 not found: ID does not exist" containerID="965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67" Oct 30 02:04:52.048296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.048261 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67"} err="failed to get container status \"965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67\": rpc error: code = NotFound desc = could not find container \"965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67\": container with ID starting with 965dd4c76e5052b06567924430a40cb50fe5c18efc740fc036b762caede3fd67 not found: ID does not exist" Oct 30 02:04:52.083605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.083559 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl"] Oct 30 02:04:52.087414 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:52.087378 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-l86vl"] Oct 30 02:04:53.092077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:53.092036 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr"] Oct 30 02:04:53.130289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:53.130247 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27"] Oct 30 02:04:53.644264 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:53.644229 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" path="/var/lib/kubelet/pods/1b81a0ab-3b26-403b-bfb3-b36cf060240f/volumes" Oct 30 02:04:58.097551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.094091 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" podUID="40658182-48e2-4357-83a9-a66778baa6b8" containerName="payload-processing" containerID="cri-o://4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5" gracePeriod=25 Oct 30 02:04:58.142606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.137950 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" podUID="0c6bcae2-5da9-40ce-9f94-def87a42ac3f" containerName="payload-pre-processing" containerID="cri-o://a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36" gracePeriod=25 Oct 30 02:04:58.596584 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.596181 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:04:58.606836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.606776 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:04:58.724557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.724496 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40658182-48e2-4357-83a9-a66778baa6b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40658182-48e2-4357-83a9-a66778baa6b8-plugins-config-volume\") pod \"40658182-48e2-4357-83a9-a66778baa6b8\" (UID: \"40658182-48e2-4357-83a9-a66778baa6b8\") " Oct 30 02:04:58.724762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.724603 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-kube-api-access-g7jlh\" (UniqueName: \"kubernetes.io/projected/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-kube-api-access-g7jlh\") pod \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\" (UID: \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\") " Oct 30 02:04:58.724762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.724736 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40658182-48e2-4357-83a9-a66778baa6b8-kube-api-access-rq6wg\" (UniqueName: \"kubernetes.io/projected/40658182-48e2-4357-83a9-a66778baa6b8-kube-api-access-rq6wg\") pod \"40658182-48e2-4357-83a9-a66778baa6b8\" (UID: \"40658182-48e2-4357-83a9-a66778baa6b8\") " Oct 30 02:04:58.724882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.724824 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-plugins-config-volume\") pod \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\" (UID: \"0c6bcae2-5da9-40ce-9f94-def87a42ac3f\") " Oct 30 02:04:58.725040 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.725007 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40658182-48e2-4357-83a9-a66778baa6b8-plugins-config-volume" pod "40658182-48e2-4357-83a9-a66778baa6b8" (UID: "40658182-48e2-4357-83a9-a66778baa6b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:58.725516 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.725491 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40658182-48e2-4357-83a9-a66778baa6b8-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:58.725829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.725804 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-plugins-config-volume" pod "0c6bcae2-5da9-40ce-9f94-def87a42ac3f" (UID: "0c6bcae2-5da9-40ce-9f94-def87a42ac3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:58.727896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.727854 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40658182-48e2-4357-83a9-a66778baa6b8-kube-api-access-rq6wg" pod "40658182-48e2-4357-83a9-a66778baa6b8" (UID: "40658182-48e2-4357-83a9-a66778baa6b8"). InnerVolumeSpecName "kube-api-access-rq6wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:58.727994 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.727912 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-kube-api-access-g7jlh" pod "0c6bcae2-5da9-40ce-9f94-def87a42ac3f" (UID: "0c6bcae2-5da9-40ce-9f94-def87a42ac3f"). InnerVolumeSpecName "kube-api-access-g7jlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:58.826794 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.826695 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:58.826794 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.826736 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7jlh\" (UniqueName: \"kubernetes.io/projected/0c6bcae2-5da9-40ce-9f94-def87a42ac3f-kube-api-access-g7jlh\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:58.826794 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:58.826749 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq6wg\" (UniqueName: \"kubernetes.io/projected/40658182-48e2-4357-83a9-a66778baa6b8-kube-api-access-rq6wg\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:04:59.118048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.117931 5035 generic.go:356] "Generic (PLEG): container finished" podID="40658182-48e2-4357-83a9-a66778baa6b8" containerID="4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5" exitCode=0 Oct 30 02:04:59.118048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.118000 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" Oct 30 02:04:59.118694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.118115 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" event={"ID":"40658182-48e2-4357-83a9-a66778baa6b8","Type":"ContainerDied","Data":"4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5"} Oct 30 02:04:59.118694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.118169 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr" event={"ID":"40658182-48e2-4357-83a9-a66778baa6b8","Type":"ContainerDied","Data":"ee2f57031d788adfccf887bbca4d0fca1197c1aa2be15bb0796f588b061cdfe6"} Oct 30 02:04:59.118694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.118201 5035 scope.go:122] "RemoveContainer" containerID="4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5" Oct 30 02:04:59.121862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.121828 5035 generic.go:356] "Generic (PLEG): container finished" podID="0c6bcae2-5da9-40ce-9f94-def87a42ac3f" containerID="a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36" exitCode=0 Oct 30 02:04:59.122183 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.121863 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" Oct 30 02:04:59.122452 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.121886 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" event={"ID":"0c6bcae2-5da9-40ce-9f94-def87a42ac3f","Type":"ContainerDied","Data":"a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36"} Oct 30 02:04:59.122652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.122587 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27" event={"ID":"0c6bcae2-5da9-40ce-9f94-def87a42ac3f","Type":"ContainerDied","Data":"7f734409bc68c5ae5d3127af1a5cf2a3aa70ded975f5ca5300fe88dd92439c03"} Oct 30 02:04:59.140043 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.138072 5035 scope.go:122] "RemoveContainer" containerID="4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5" Oct 30 02:04:59.140043 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:59.138466 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5\": container with ID starting with 4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5 not found: ID does not exist" containerID="4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5" Oct 30 02:04:59.140043 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.138512 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5"} err="failed to get container status \"4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5\": rpc error: code = NotFound desc = could not find container \"4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5\": container with ID starting with 4280139bef4411ebe4de9690c7444ddb005e4c57a091de1c5c77b47be7ecc5d5 not found: ID does not exist" Oct 30 02:04:59.140043 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.138565 5035 scope.go:122] "RemoveContainer" containerID="a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36" Oct 30 02:04:59.161782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.161757 5035 scope.go:122] "RemoveContainer" containerID="a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36" Oct 30 02:04:59.162139 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:04:59.162117 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36\": container with ID starting with a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36 not found: ID does not exist" containerID="a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36" Oct 30 02:04:59.162244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.162147 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36"} err="failed to get container status \"a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36\": rpc error: code = NotFound desc = could not find container \"a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36\": container with ID starting with a015f1530725cc6a55ecfb053c65fcc5dcf5f87eab8938adcde13bfccba4ac36 not found: ID does not exist" Oct 30 02:04:59.210553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.209601 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27"] Oct 30 02:04:59.218083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.216303 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-003ec6-84bbd7b97f-n8t27"] Oct 30 02:04:59.243224 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.243179 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr"] Oct 30 02:04:59.248883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.248846 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-003ec6-58c5d44fbd-m47lr"] Oct 30 02:04:59.644633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.644596 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6bcae2-5da9-40ce-9f94-def87a42ac3f" path="/var/lib/kubelet/pods/0c6bcae2-5da9-40ce-9f94-def87a42ac3f/volumes" Oct 30 02:04:59.645080 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:04:59.645066 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40658182-48e2-4357-83a9-a66778baa6b8" path="/var/lib/kubelet/pods/40658182-48e2-4357-83a9-a66778baa6b8/volumes" Oct 30 02:05:01.473830 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:01.473787 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm"] Oct 30 02:05:01.474417 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:01.474123 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" containerID="cri-o://578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc" gracePeriod=30 Oct 30 02:05:06.774212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.774174 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:05:06.953766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.953722 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-token\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-token\") pod \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " Oct 30 02:05:06.953953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.953781 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-kube-api-access-t2ccg\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-kube-api-access-t2ccg\") pod \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " Oct 30 02:05:06.953953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.953824 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-socket\") pod \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " Oct 30 02:05:06.953953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.953861 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-data\") pod \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " Oct 30 02:05:06.953953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.953902 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-certs\") pod \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " Oct 30 02:05:06.953953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.953949 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istiod-ca-cert\") pod \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " Oct 30 02:05:06.954215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.954133 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-envoy\") pod \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " Oct 30 02:05:06.954215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.954142 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-socket" pod "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" (UID: "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:06.954315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.954224 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-credential-socket\") pod \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " Oct 30 02:05:06.954394 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.954373 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-podinfo\") pod \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\" (UID: \"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c\") " Oct 30 02:05:06.954455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.954435 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-data" pod "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" (UID: "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:06.954758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.954521 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istiod-ca-cert" pod "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" (UID: "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:06.954758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.954619 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-credential-socket" pod "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" (UID: "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:06.954758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.954689 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-certs" pod "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" (UID: "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:06.955050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.955026 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:06.955117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.955057 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:06.955117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.955074 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:06.955117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.955089 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:06.955117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.955104 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:06.956353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.956325 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-token" pod "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" (UID: "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:06.956511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.956465 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-kube-api-access-t2ccg" pod "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" (UID: "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c"). InnerVolumeSpecName "kube-api-access-t2ccg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:06.956723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.956689 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-envoy" pod "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" (UID: "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:06.956835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:06.956820 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-podinfo" pod "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" (UID: "c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:05:07.058165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.058056 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:07.058165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.058095 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:07.058165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.058110 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2ccg\" (UniqueName: \"kubernetes.io/projected/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-kube-api-access-t2ccg\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:07.058165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.058120 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:07.186540 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.186486 5035 generic.go:356] "Generic (PLEG): container finished" podID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerID="578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc" exitCode=0 Oct 30 02:05:07.186733 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.186562 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" event={"ID":"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c","Type":"ContainerDied","Data":"578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc"} Oct 30 02:05:07.186733 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.186618 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" event={"ID":"c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c","Type":"ContainerDied","Data":"9e89d1806fe89b422abd622f810b8869bb531e4bef186d28b57dfd72f9f2f7fc"} Oct 30 02:05:07.186733 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.186643 5035 scope.go:122] "RemoveContainer" containerID="578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc" Oct 30 02:05:07.186733 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.186567 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm" Oct 30 02:05:07.199118 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.199097 5035 scope.go:122] "RemoveContainer" containerID="a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0" Oct 30 02:05:07.217021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.216995 5035 scope.go:122] "RemoveContainer" containerID="578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc" Oct 30 02:05:07.217473 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:05:07.217436 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc\": container with ID starting with 578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc not found: ID does not exist" containerID="578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc" Oct 30 02:05:07.217669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.217484 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc"} err="failed to get container status \"578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc\": rpc error: code = NotFound desc = could not find container \"578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc\": container with ID starting with 578f83762788d475ee0edcc35a868c1dda5ffc0fe0c772ba03d541053357e8dc not found: ID does not exist" Oct 30 02:05:07.217669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.217512 5035 scope.go:122] "RemoveContainer" containerID="a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0" Oct 30 02:05:07.217911 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:05:07.217886 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0\": container with ID starting with a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0 not found: ID does not exist" containerID="a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0" Oct 30 02:05:07.217983 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.217921 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0"} err="failed to get container status \"a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0\": rpc error: code = NotFound desc = could not find container \"a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0\": container with ID starting with a692662cf1e3329b2fbb252ce256c4146816a5e8edac117115896a46c37a67a0 not found: ID does not exist" Oct 30 02:05:07.245872 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.245837 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm"] Oct 30 02:05:07.249450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.249414 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-003ec6-openshift-default-74dcb8dd9-kw6rm"] Oct 30 02:05:07.645594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:07.645557 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" path="/var/lib/kubelet/pods/c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c/volumes" Oct 30 02:05:12.829440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:12.829397 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv"] Oct 30 02:05:12.830333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:12.830283 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" podUID="4eac47eb-462c-49dd-8d15-cd6f90dac721" containerName="maas-api" containerID="cri-o://8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b" gracePeriod=30 Oct 30 02:05:13.164167 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.164139 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 02:05:13.236440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.236405 5035 generic.go:356] "Generic (PLEG): container finished" podID="4eac47eb-462c-49dd-8d15-cd6f90dac721" containerID="8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b" exitCode=0 Oct 30 02:05:13.236689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.236449 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" event={"ID":"4eac47eb-462c-49dd-8d15-cd6f90dac721","Type":"ContainerDied","Data":"8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b"} Oct 30 02:05:13.236689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.236465 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" Oct 30 02:05:13.236689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.236492 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv" event={"ID":"4eac47eb-462c-49dd-8d15-cd6f90dac721","Type":"ContainerDied","Data":"57e11dee994d4924b083e8844e90eca9723ca9cf2c7910869b0b02233badf0be"} Oct 30 02:05:13.236689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.236513 5035 scope.go:122] "RemoveContainer" containerID="8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b" Oct 30 02:05:13.245971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.245937 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-metrics-certs\") pod \"4eac47eb-462c-49dd-8d15-cd6f90dac721\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " Oct 30 02:05:13.246710 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.246030 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eac47eb-462c-49dd-8d15-cd6f90dac721-kube-api-access-kqmv9\" (UniqueName: \"kubernetes.io/projected/4eac47eb-462c-49dd-8d15-cd6f90dac721-kube-api-access-kqmv9\") pod \"4eac47eb-462c-49dd-8d15-cd6f90dac721\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " Oct 30 02:05:13.246710 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.246107 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eac47eb-462c-49dd-8d15-cd6f90dac721-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eac47eb-462c-49dd-8d15-cd6f90dac721-openshift-service-ca\") pod \"4eac47eb-462c-49dd-8d15-cd6f90dac721\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " Oct 30 02:05:13.246710 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.246222 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-maas-api-tls\") pod \"4eac47eb-462c-49dd-8d15-cd6f90dac721\" (UID: \"4eac47eb-462c-49dd-8d15-cd6f90dac721\") " Oct 30 02:05:13.247630 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.247343 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eac47eb-462c-49dd-8d15-cd6f90dac721-openshift-service-ca" pod "4eac47eb-462c-49dd-8d15-cd6f90dac721" (UID: "4eac47eb-462c-49dd-8d15-cd6f90dac721"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:13.249435 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.249406 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-maas-api-tls" pod "4eac47eb-462c-49dd-8d15-cd6f90dac721" (UID: "4eac47eb-462c-49dd-8d15-cd6f90dac721"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:13.250618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.250573 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-metrics-certs" pod "4eac47eb-462c-49dd-8d15-cd6f90dac721" (UID: "4eac47eb-462c-49dd-8d15-cd6f90dac721"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:13.251314 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.251294 5035 scope.go:122] "RemoveContainer" containerID="8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b" Oct 30 02:05:13.251947 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:05:13.251911 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b\": container with ID starting with 8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b not found: ID does not exist" containerID="8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b" Oct 30 02:05:13.252053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.251955 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b"} err="failed to get container status \"8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b\": rpc error: code = NotFound desc = could not find container \"8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b\": container with ID starting with 8eca94d0c8417aa33c6c978cd5bfe5bc5866433f6440cbb416df0dc44587861b not found: ID does not exist" Oct 30 02:05:13.253073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.253044 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eac47eb-462c-49dd-8d15-cd6f90dac721-kube-api-access-kqmv9" pod "4eac47eb-462c-49dd-8d15-cd6f90dac721" (UID: "4eac47eb-462c-49dd-8d15-cd6f90dac721"). InnerVolumeSpecName "kube-api-access-kqmv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:13.372554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.364872 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eac47eb-462c-49dd-8d15-cd6f90dac721-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:13.372554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.364925 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:13.372554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.364940 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eac47eb-462c-49dd-8d15-cd6f90dac721-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:13.372554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.364956 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqmv9\" (UniqueName: \"kubernetes.io/projected/4eac47eb-462c-49dd-8d15-cd6f90dac721-kube-api-access-kqmv9\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:13.588605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.588560 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv"] Oct 30 02:05:13.601721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.601683 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5c90f5-576f4d9b7-fh4mv"] Oct 30 02:05:13.644037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:13.643997 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eac47eb-462c-49dd-8d15-cd6f90dac721" path="/var/lib/kubelet/pods/4eac47eb-462c-49dd-8d15-cd6f90dac721/volumes" Oct 30 02:05:17.630405 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:05:17.630366 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:05:28.194890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.194788 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh"] Oct 30 02:05:28.195834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195802 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" Oct 30 02:05:28.195834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195833 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195852 5035 cpu_manager.go:414] "containerMap: removing container" podUID="40658182-48e2-4357-83a9-a66778baa6b8" containerName="payload-processing" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195870 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40658182-48e2-4357-83a9-a66778baa6b8" containerName="payload-processing" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195891 5035 cpu_manager.go:414] "containerMap: removing container" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195900 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195928 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0c6bcae2-5da9-40ce-9f94-def87a42ac3f" containerName="payload-pre-processing" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195936 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6bcae2-5da9-40ce-9f94-def87a42ac3f" containerName="payload-pre-processing" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195950 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.195958 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196013 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196022 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196035 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" Oct 30 02:05:28.196042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196043 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196054 5035 cpu_manager.go:414] "containerMap: removing container" podUID="47c22c39-4909-447a-a55b-c20294cdcd1f" containerName="payload-processing" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196061 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c22c39-4909-447a-a55b-c20294cdcd1f" containerName="payload-processing" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196080 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c58096a8-ee94-4650-b166-ff870a6b5f08" containerName="payload-pre-processing" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196087 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58096a8-ee94-4650-b166-ff870a6b5f08" containerName="payload-pre-processing" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196107 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196116 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196131 5035 cpu_manager.go:414] "containerMap: removing container" podUID="4eac47eb-462c-49dd-8d15-cd6f90dac721" containerName="maas-api" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196138 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eac47eb-462c-49dd-8d15-cd6f90dac721" containerName="maas-api" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196169 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196177 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196337 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196352 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196362 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="40658182-48e2-4357-83a9-a66778baa6b8" containerName="payload-processing" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196373 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="24815994-57e7-4a2a-9a36-0be361ef7ba1" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196385 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6bcae2-5da9-40ce-9f94-def87a42ac3f" containerName="payload-pre-processing" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196398 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bdac7d-3ef7-4906-ac0f-57f33ec5a08c" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196410 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196421 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c22c39-4909-447a-a55b-c20294cdcd1f" containerName="payload-processing" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196431 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58096a8-ee94-4650-b166-ff870a6b5f08" containerName="payload-pre-processing" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196443 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fb784e-0d01-4def-bdc1-63259c9e1fbf" containerName="istio-proxy" Oct 30 02:05:28.196744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.196452 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eac47eb-462c-49dd-8d15-cd6f90dac721" containerName="maas-api" Oct 30 02:05:28.202757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.202728 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.225339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.225258 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh"] Oct 30 02:05:28.305854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.305813 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.306123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.306094 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.306168 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.306137 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hhv6\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-kube-api-access-2hhv6\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.306205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.306179 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.306300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.306286 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.306384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.306370 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.306423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.306412 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.306566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.306553 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.306624 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.306588 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.407668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.407630 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.407837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.407697 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.407837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.407816 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.407913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.407869 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.407981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.407917 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.407981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.407944 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.407981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.407970 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhv6\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-kube-api-access-2hhv6\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.408141 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.408027 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.408141 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.408120 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.408273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.408244 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.408273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.408264 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.408430 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.408376 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.408521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.408502 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.408912 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.408885 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.410231 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.410195 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.410354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.410335 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.433609 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.433563 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhv6\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-kube-api-access-2hhv6\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.433927 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.433900 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.524468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.524410 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:28.853657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.853629 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh"] Oct 30 02:05:28.855220 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:05:28.855184 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae44149_6c30_41ae_bfc1_8c07dbc642a0.slice/crio-12d08e47e99cd8340ef3502cecd7f5d6283f5a7451027dbcf0f56ff43b59ca4e WatchSource:0}: Error finding container 12d08e47e99cd8340ef3502cecd7f5d6283f5a7451027dbcf0f56ff43b59ca4e: Status 404 returned error can't find the container with id 12d08e47e99cd8340ef3502cecd7f5d6283f5a7451027dbcf0f56ff43b59ca4e Oct 30 02:05:28.857987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.857948 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:05:28.858127 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.858032 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:05:28.858127 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:28.858079 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:05:29.375468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:29.375424 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" event={"ID":"aae44149-6c30-41ae-bfc1-8c07dbc642a0","Type":"ContainerStarted","Data":"839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b"} Oct 30 02:05:29.375917 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:29.375488 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" event={"ID":"aae44149-6c30-41ae-bfc1-8c07dbc642a0","Type":"ContainerStarted","Data":"12d08e47e99cd8340ef3502cecd7f5d6283f5a7451027dbcf0f56ff43b59ca4e"} Oct 30 02:05:29.431435 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:29.431371 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" podStartSLOduration=1.431354148 podStartE2EDuration="1.431354148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:29.427574576 +0000 UTC m=+2207.930506804" watchObservedRunningTime="2026-10-30 02:05:29.431354148 +0000 UTC m=+2207.934286449" Oct 30 02:05:29.524634 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:29.524598 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:29.526403 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:29.526374 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:29.526519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:29.526429 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 02:05:30.525856 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:30.525814 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:30.526356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:30.525884 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 02:05:31.525933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:31.525887 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:31.526803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:31.525949 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 02:05:32.525462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:32.525414 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:32.525722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:32.525485 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 02:05:33.525816 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:33.525768 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:33.526302 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:33.525852 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 02:05:34.525121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:34.525065 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:34.525328 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:34.525127 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 02:05:35.525304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:35.525261 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 02:05:35.525890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:35.525330 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 02:05:36.530281 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:36.530245 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:36.530724 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:36.530561 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:36.531257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:36.531236 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:05:42.565677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.565632 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8"] Oct 30 02:05:42.566357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.565885 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" podUID="d7fb44dd-ba03-4128-967d-bfaf2fc21090" containerName="maas-api" containerID="cri-o://67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26" gracePeriod=30 Oct 30 02:05:42.890600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.890569 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:05:42.959421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.959371 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-maas-api-tls\") pod \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " Oct 30 02:05:42.959665 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.959585 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7fb44dd-ba03-4128-967d-bfaf2fc21090-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7fb44dd-ba03-4128-967d-bfaf2fc21090-openshift-service-ca\") pod \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " Oct 30 02:05:42.959738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.959726 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-metrics-certs\") pod \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " Oct 30 02:05:42.959833 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.959816 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7fb44dd-ba03-4128-967d-bfaf2fc21090-kube-api-access-b6drj\" (UniqueName: \"kubernetes.io/projected/d7fb44dd-ba03-4128-967d-bfaf2fc21090-kube-api-access-b6drj\") pod \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\" (UID: \"d7fb44dd-ba03-4128-967d-bfaf2fc21090\") " Oct 30 02:05:42.960098 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.960067 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7fb44dd-ba03-4128-967d-bfaf2fc21090-openshift-service-ca" pod "d7fb44dd-ba03-4128-967d-bfaf2fc21090" (UID: "d7fb44dd-ba03-4128-967d-bfaf2fc21090"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:42.960882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.960852 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7fb44dd-ba03-4128-967d-bfaf2fc21090-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:42.962679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.962639 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-metrics-certs" pod "d7fb44dd-ba03-4128-967d-bfaf2fc21090" (UID: "d7fb44dd-ba03-4128-967d-bfaf2fc21090"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:42.962888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.962857 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-maas-api-tls" pod "d7fb44dd-ba03-4128-967d-bfaf2fc21090" (UID: "d7fb44dd-ba03-4128-967d-bfaf2fc21090"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:42.963219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:42.963195 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7fb44dd-ba03-4128-967d-bfaf2fc21090-kube-api-access-b6drj" pod "d7fb44dd-ba03-4128-967d-bfaf2fc21090" (UID: "d7fb44dd-ba03-4128-967d-bfaf2fc21090"). InnerVolumeSpecName "kube-api-access-b6drj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:43.063070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.062985 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:43.063070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.063031 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7fb44dd-ba03-4128-967d-bfaf2fc21090-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:43.063070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.063049 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6drj\" (UniqueName: \"kubernetes.io/projected/d7fb44dd-ba03-4128-967d-bfaf2fc21090-kube-api-access-b6drj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:43.347467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.347430 5035 scope.go:122] "RemoveContainer" containerID="15fafa42409166c76b7e9e6b91383f846c5e4e1d5015e4ba239c3fca8c6cc51a" Oct 30 02:05:43.358950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.358926 5035 scope.go:122] "RemoveContainer" containerID="b42c92fc1b1d26f61912b82e30612d97e6b7f9d235d489df9b43deb1cd5d7ec0" Oct 30 02:05:43.498764 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.498725 5035 generic.go:356] "Generic (PLEG): container finished" podID="d7fb44dd-ba03-4128-967d-bfaf2fc21090" containerID="67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26" exitCode=0 Oct 30 02:05:43.498984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.498795 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" Oct 30 02:05:43.498984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.498809 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" event={"ID":"d7fb44dd-ba03-4128-967d-bfaf2fc21090","Type":"ContainerDied","Data":"67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26"} Oct 30 02:05:43.498984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.498904 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8" event={"ID":"d7fb44dd-ba03-4128-967d-bfaf2fc21090","Type":"ContainerDied","Data":"cdd64a2eda95ba13e4330af356ca31e514d5637acb5ce5afe76b9b73e47b30dc"} Oct 30 02:05:43.498984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.498940 5035 scope.go:122] "RemoveContainer" containerID="67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26" Oct 30 02:05:43.510257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.510235 5035 scope.go:122] "RemoveContainer" containerID="67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26" Oct 30 02:05:43.510640 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:05:43.510615 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26\": container with ID starting with 67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26 not found: ID does not exist" containerID="67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26" Oct 30 02:05:43.510741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.510653 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26"} err="failed to get container status \"67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26\": rpc error: code = NotFound desc = could not find container \"67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26\": container with ID starting with 67f7cf4ad915079f4c0bf4f90f630f1d41ceb39a891e77086cb0cfbeaf4a4c26 not found: ID does not exist" Oct 30 02:05:43.569624 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.569091 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8"] Oct 30 02:05:43.583446 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.583401 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-145336-786d75f756-8jsp8"] Oct 30 02:05:43.645065 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:43.644969 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7fb44dd-ba03-4128-967d-bfaf2fc21090" path="/var/lib/kubelet/pods/d7fb44dd-ba03-4128-967d-bfaf2fc21090/volumes" Oct 30 02:05:47.633882 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:05:47.633824 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:05:53.412952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:53.412881 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx"] Oct 30 02:05:53.413681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:53.413215 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" containerID="cri-o://8d79d33b355339578111d585e523fa4901f916f015a3a64b69b3cc3345626c12" gracePeriod=30 Oct 30 02:05:54.691441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:54.691397 5035 patch_prober.go:29] interesting pod/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:05:54.691964 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:54.691484 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:05:58.625123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.625025 5035 generic.go:356] "Generic (PLEG): container finished" podID="0a0da372-b63d-4050-899c-eb5007647157" containerID="8d79d33b355339578111d585e523fa4901f916f015a3a64b69b3cc3345626c12" exitCode=0 Oct 30 02:05:58.625878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.625086 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" event={"ID":"0a0da372-b63d-4050-899c-eb5007647157","Type":"ContainerDied","Data":"8d79d33b355339578111d585e523fa4901f916f015a3a64b69b3cc3345626c12"} Oct 30 02:05:58.626078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.626064 5035 scope.go:122] "RemoveContainer" containerID="145e6d1c1b67436098570cd37cee3ada549268458362fd9c43fe0ad96095a0e5" Oct 30 02:05:58.705857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.705825 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:05:58.736204 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736170 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-kube-api-access-v2tqd\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-kube-api-access-v2tqd\") pod \"0a0da372-b63d-4050-899c-eb5007647157\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " Oct 30 02:05:58.736421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736274 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-data\") pod \"0a0da372-b63d-4050-899c-eb5007647157\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " Oct 30 02:05:58.736421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736299 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-envoy\") pod \"0a0da372-b63d-4050-899c-eb5007647157\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " Oct 30 02:05:58.736421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736329 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-certs\") pod \"0a0da372-b63d-4050-899c-eb5007647157\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " Oct 30 02:05:58.736651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736452 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-istio-token\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-istio-token\") pod \"0a0da372-b63d-4050-899c-eb5007647157\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " Oct 30 02:05:58.736651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736496 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-socket\") pod \"0a0da372-b63d-4050-899c-eb5007647157\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " Oct 30 02:05:58.736651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736564 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a0da372-b63d-4050-899c-eb5007647157-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0da372-b63d-4050-899c-eb5007647157-istio-podinfo\") pod \"0a0da372-b63d-4050-899c-eb5007647157\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " Oct 30 02:05:58.736651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736636 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0da372-b63d-4050-899c-eb5007647157-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0da372-b63d-4050-899c-eb5007647157-istiod-ca-cert\") pod \"0a0da372-b63d-4050-899c-eb5007647157\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " Oct 30 02:05:58.736871 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736680 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-credential-socket\") pod \"0a0da372-b63d-4050-899c-eb5007647157\" (UID: \"0a0da372-b63d-4050-899c-eb5007647157\") " Oct 30 02:05:58.736871 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736707 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-data" pod "0a0da372-b63d-4050-899c-eb5007647157" (UID: "0a0da372-b63d-4050-899c-eb5007647157"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:58.736871 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736765 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-certs" pod "0a0da372-b63d-4050-899c-eb5007647157" (UID: "0a0da372-b63d-4050-899c-eb5007647157"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:58.737039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.736923 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-socket" pod "0a0da372-b63d-4050-899c-eb5007647157" (UID: "0a0da372-b63d-4050-899c-eb5007647157"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:58.737100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.737051 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-credential-socket" pod "0a0da372-b63d-4050-899c-eb5007647157" (UID: "0a0da372-b63d-4050-899c-eb5007647157"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:58.737100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.737079 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0da372-b63d-4050-899c-eb5007647157-istiod-ca-cert" pod "0a0da372-b63d-4050-899c-eb5007647157" (UID: "0a0da372-b63d-4050-899c-eb5007647157"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:58.737379 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.737351 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.737379 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.737376 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0da372-b63d-4050-899c-eb5007647157-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.737572 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.737390 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.737572 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.737405 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.737572 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.737419 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.738665 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.738636 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-envoy" pod "0a0da372-b63d-4050-899c-eb5007647157" (UID: "0a0da372-b63d-4050-899c-eb5007647157"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:58.738797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.738690 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-kube-api-access-v2tqd" pod "0a0da372-b63d-4050-899c-eb5007647157" (UID: "0a0da372-b63d-4050-899c-eb5007647157"). InnerVolumeSpecName "kube-api-access-v2tqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:58.738916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.738893 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a0da372-b63d-4050-899c-eb5007647157-istio-podinfo" pod "0a0da372-b63d-4050-899c-eb5007647157" (UID: "0a0da372-b63d-4050-899c-eb5007647157"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:05:58.739124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.739096 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-istio-token" pod "0a0da372-b63d-4050-899c-eb5007647157" (UID: "0a0da372-b63d-4050-899c-eb5007647157"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:58.838540 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.838499 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0da372-b63d-4050-899c-eb5007647157-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.838540 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.838547 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.838770 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.838556 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0da372-b63d-4050-899c-eb5007647157-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.838770 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:58.838565 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tqd\" (UniqueName: \"kubernetes.io/projected/0a0da372-b63d-4050-899c-eb5007647157-kube-api-access-v2tqd\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:05:59.653726 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:59.653682 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" event={"ID":"0a0da372-b63d-4050-899c-eb5007647157","Type":"ContainerDied","Data":"ba654e83010589dfaf0f810be3ca17d636f8d55d7114914b2c3c84c291693526"} Oct 30 02:05:59.654333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:59.653760 5035 scope.go:122] "RemoveContainer" containerID="8d79d33b355339578111d585e523fa4901f916f015a3a64b69b3cc3345626c12" Oct 30 02:05:59.654333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:59.653896 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx" Oct 30 02:05:59.736623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:59.736582 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx"] Oct 30 02:05:59.741458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:05:59.741420 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-145336-openshift-default-8657b6985c-dkvjx"] Oct 30 02:06:01.357571 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.357483 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst"] Oct 30 02:06:01.358479 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.358451 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d7fb44dd-ba03-4128-967d-bfaf2fc21090" containerName="maas-api" Oct 30 02:06:01.358637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.358481 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7fb44dd-ba03-4128-967d-bfaf2fc21090" containerName="maas-api" Oct 30 02:06:01.358637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.358521 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" Oct 30 02:06:01.358637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.358546 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" Oct 30 02:06:01.358819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.358751 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" Oct 30 02:06:01.358819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.358773 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7fb44dd-ba03-4128-967d-bfaf2fc21090" containerName="maas-api" Oct 30 02:06:01.358819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.358785 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b81a0ab-3b26-403b-bfb3-b36cf060240f" containerName="istio-proxy" Oct 30 02:06:01.358819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.358795 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" Oct 30 02:06:01.385942 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.364829 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:01.415646 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.415605 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-428529\"" Oct 30 02:06:01.415876 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.415800 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-428529-dockercfg-8wng5\"" Oct 30 02:06:01.425856 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.425814 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst"] Oct 30 02:06:01.502099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.502048 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ee0a8c-6993-4316-94c4-fa91b3db48db-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst\" (UID: \"42ee0a8c-6993-4316-94c4-fa91b3db48db\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:01.502339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.502246 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4kk\" (UniqueName: \"kubernetes.io/projected/42ee0a8c-6993-4316-94c4-fa91b3db48db-kube-api-access-nv4kk\") pod \"payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst\" (UID: \"42ee0a8c-6993-4316-94c4-fa91b3db48db\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:01.603464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.603423 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ee0a8c-6993-4316-94c4-fa91b3db48db-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst\" (UID: \"42ee0a8c-6993-4316-94c4-fa91b3db48db\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:01.603714 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.603512 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4kk\" (UniqueName: \"kubernetes.io/projected/42ee0a8c-6993-4316-94c4-fa91b3db48db-kube-api-access-nv4kk\") pod \"payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst\" (UID: \"42ee0a8c-6993-4316-94c4-fa91b3db48db\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:01.604226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.604205 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ee0a8c-6993-4316-94c4-fa91b3db48db-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst\" (UID: \"42ee0a8c-6993-4316-94c4-fa91b3db48db\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:01.628619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.628514 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4kk\" (UniqueName: \"kubernetes.io/projected/42ee0a8c-6993-4316-94c4-fa91b3db48db-kube-api-access-nv4kk\") pod \"payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst\" (UID: \"42ee0a8c-6993-4316-94c4-fa91b3db48db\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:01.645159 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.645118 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0da372-b63d-4050-899c-eb5007647157" path="/var/lib/kubelet/pods/0a0da372-b63d-4050-899c-eb5007647157/volumes" Oct 30 02:06:01.753270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:01.753228 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:02.065368 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:02.065339 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst"] Oct 30 02:06:02.066579 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:06:02.066504 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ee0a8c_6993_4316_94c4_fa91b3db48db.slice/crio-06a334f77a73122a32d0a06b37ba3b53bae2c75190d5c45020d1b77789432231 WatchSource:0}: Error finding container 06a334f77a73122a32d0a06b37ba3b53bae2c75190d5c45020d1b77789432231: Status 404 returned error can't find the container with id 06a334f77a73122a32d0a06b37ba3b53bae2c75190d5c45020d1b77789432231 Oct 30 02:06:02.677041 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:02.676999 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" event={"ID":"42ee0a8c-6993-4316-94c4-fa91b3db48db","Type":"ContainerStarted","Data":"e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87"} Oct 30 02:06:02.677469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:02.677061 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" event={"ID":"42ee0a8c-6993-4316-94c4-fa91b3db48db","Type":"ContainerStarted","Data":"06a334f77a73122a32d0a06b37ba3b53bae2c75190d5c45020d1b77789432231"} Oct 30 02:06:02.677469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:02.677088 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:02.725391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:02.725327 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" podStartSLOduration=1.725310675 podStartE2EDuration="1.725310675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:02.724094546 +0000 UTC m=+2241.227026766" watchObservedRunningTime="2026-10-30 02:06:02.725310675 +0000 UTC m=+2241.228242900" Oct 30 02:06:05.823008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:05.822961 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5"] Oct 30 02:06:05.823602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:05.823275 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" podUID="1a9f5a20-1d2c-4b31-a37c-d049557b35d7" containerName="maas-api" containerID="cri-o://010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc" gracePeriod=30 Oct 30 02:06:06.151752 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.151719 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:06:06.290575 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.290485 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-maas-api-tls\") pod \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " Oct 30 02:06:06.290777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.290601 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-openshift-service-ca\") pod \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " Oct 30 02:06:06.290831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.290780 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-metrics-certs\") pod \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " Oct 30 02:06:06.290892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.290826 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-kube-api-access-tb8g6\" (UniqueName: \"kubernetes.io/projected/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-kube-api-access-tb8g6\") pod \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\" (UID: \"1a9f5a20-1d2c-4b31-a37c-d049557b35d7\") " Oct 30 02:06:06.291020 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.290988 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-openshift-service-ca" pod "1a9f5a20-1d2c-4b31-a37c-d049557b35d7" (UID: "1a9f5a20-1d2c-4b31-a37c-d049557b35d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:06.291428 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.291368 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:06:06.293029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.292993 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-maas-api-tls" pod "1a9f5a20-1d2c-4b31-a37c-d049557b35d7" (UID: "1a9f5a20-1d2c-4b31-a37c-d049557b35d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:06.293153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.293118 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-kube-api-access-tb8g6" pod "1a9f5a20-1d2c-4b31-a37c-d049557b35d7" (UID: "1a9f5a20-1d2c-4b31-a37c-d049557b35d7"). InnerVolumeSpecName "kube-api-access-tb8g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:06.293305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.293281 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-metrics-certs" pod "1a9f5a20-1d2c-4b31-a37c-d049557b35d7" (UID: "1a9f5a20-1d2c-4b31-a37c-d049557b35d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:06.396965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.396861 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:06:06.396965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.396913 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb8g6\" (UniqueName: \"kubernetes.io/projected/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-kube-api-access-tb8g6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:06:06.396965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.396930 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f5a20-1d2c-4b31-a37c-d049557b35d7-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:06:06.737576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.737498 5035 generic.go:356] "Generic (PLEG): container finished" podID="1a9f5a20-1d2c-4b31-a37c-d049557b35d7" containerID="010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc" exitCode=0 Oct 30 02:06:06.737808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.737628 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" Oct 30 02:06:06.737808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.737658 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" event={"ID":"1a9f5a20-1d2c-4b31-a37c-d049557b35d7","Type":"ContainerDied","Data":"010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc"} Oct 30 02:06:06.737808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.737728 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5" event={"ID":"1a9f5a20-1d2c-4b31-a37c-d049557b35d7","Type":"ContainerDied","Data":"d741deb7b191e5030106f71e7fc2bf9b9620fd97c8700edb18eee9cd90186425"} Oct 30 02:06:06.737808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.737759 5035 scope.go:122] "RemoveContainer" containerID="010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc" Oct 30 02:06:06.750404 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.750382 5035 scope.go:122] "RemoveContainer" containerID="010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc" Oct 30 02:06:06.750758 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:06:06.750736 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc\": container with ID starting with 010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc not found: ID does not exist" containerID="010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc" Oct 30 02:06:06.750839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.750771 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc"} err="failed to get container status \"010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc\": rpc error: code = NotFound desc = could not find container \"010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc\": container with ID starting with 010e0bc8913f7e6e65995283f1acdcf58f6b028efc19849df6889ac66d1f61fc not found: ID does not exist" Oct 30 02:06:06.796303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.796202 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5"] Oct 30 02:06:06.799370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:06.799347 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f3cd8-7d47bf98d8-vm6s5"] Oct 30 02:06:07.644166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:07.644130 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9f5a20-1d2c-4b31-a37c-d049557b35d7" path="/var/lib/kubelet/pods/1a9f5a20-1d2c-4b31-a37c-d049557b35d7/volumes" Oct 30 02:06:13.687126 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:13.687093 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:06:17.798974 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:06:17.798915 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:06:36.471977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.471932 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv"] Oct 30 02:06:36.472712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.472690 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" Oct 30 02:06:36.472712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.472711 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0da372-b63d-4050-899c-eb5007647157" containerName="istio-proxy" Oct 30 02:06:36.472892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.472723 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1a9f5a20-1d2c-4b31-a37c-d049557b35d7" containerName="maas-api" Oct 30 02:06:36.472892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.472729 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9f5a20-1d2c-4b31-a37c-d049557b35d7" containerName="maas-api" Oct 30 02:06:36.472892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.472850 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9f5a20-1d2c-4b31-a37c-d049557b35d7" containerName="maas-api" Oct 30 02:06:36.478794 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.476930 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.498899 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.498131 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b4a561-openshift-default-dockercfg-fbbzn\"" Oct 30 02:06:36.518723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.518678 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv"] Oct 30 02:06:36.565638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.565599 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-podinfo\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.565842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.565711 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-credential-socket\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.565842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.565806 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-certs\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.565971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.565885 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7j7\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-kube-api-access-kw7j7\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.565971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.565924 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-socket\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.566061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.566011 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istiod-ca-cert\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.566105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.566082 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-envoy\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.566147 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.566104 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-data\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.566184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.566171 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-token\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.675553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.675401 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-podinfo\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.675553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.675480 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-credential-socket\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.675854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.675601 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-certs\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.675854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.675635 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7j7\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-kube-api-access-kw7j7\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.675854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.675666 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-socket\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.675854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.675700 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istiod-ca-cert\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.675854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.675756 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-envoy\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.675854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.675782 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-data\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.676187 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.675862 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-token\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.680550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.678974 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-socket\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.680550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.679394 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-credential-socket\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.680550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.679821 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-certs\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.691553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.681284 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istiod-ca-cert\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.691553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.681622 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-data\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.695594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.694481 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-envoy\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.727608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.713736 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-podinfo\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.727608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.714661 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7j7\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-kube-api-access-kw7j7\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.727608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.721162 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-token\") pod \"e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:36.796206 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:36.796164 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:37.126835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:37.126802 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv"] Oct 30 02:06:37.128494 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:06:37.128464 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3cea51_d79d_4f51_9ebb_bd28f0335889.slice/crio-22972156251e181c71960f8619998dc83e732acdb48c17a5cdeef35eb3ca2fda WatchSource:0}: Error finding container 22972156251e181c71960f8619998dc83e732acdb48c17a5cdeef35eb3ca2fda: Status 404 returned error can't find the container with id 22972156251e181c71960f8619998dc83e732acdb48c17a5cdeef35eb3ca2fda Oct 30 02:06:37.131198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:37.131162 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:06:37.131276 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:37.131238 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:06:37.131276 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:37.131269 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:06:38.131831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:38.111601 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" event={"ID":"ef3cea51-d79d-4f51-9ebb-bd28f0335889","Type":"ContainerStarted","Data":"a395e1a73e2180088d7146685523ee09520eb874b6ae2526a22050dd337cce6b"} Oct 30 02:06:38.131831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:38.111693 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" event={"ID":"ef3cea51-d79d-4f51-9ebb-bd28f0335889","Type":"ContainerStarted","Data":"22972156251e181c71960f8619998dc83e732acdb48c17a5cdeef35eb3ca2fda"} Oct 30 02:06:38.214570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:38.207122 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" podStartSLOduration=2.207097374 podStartE2EDuration="2.207097374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:38.18405735 +0000 UTC m=+2276.686989591" watchObservedRunningTime="2026-10-30 02:06:38.207097374 +0000 UTC m=+2276.710029597" Oct 30 02:06:38.798960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:38.798870 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:38.806803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:38.806769 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:39.124576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:39.124462 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:39.126219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:39.126188 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:06:47.630258 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:06:47.630215 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:06:49.920260 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:49.920216 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh"] Oct 30 02:06:49.926186 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:49.926163 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:49.952751 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:49.952709 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh"] Oct 30 02:06:50.074153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.074109 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.074356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.074253 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.074356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.074293 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.074438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.074402 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfg74\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-kube-api-access-cfg74\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.074438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.074428 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.074505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.074446 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.074597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.074582 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.074668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.074652 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.074706 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.074679 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.176341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176227 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfg74\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-kube-api-access-cfg74\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.176341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176290 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.176341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176314 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.176746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176369 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.176746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176445 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.176746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176473 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.176746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176580 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.176746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176673 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.176746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176708 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.177072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176896 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.177072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176945 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.177072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.176976 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.177218 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.177125 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.177506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.177486 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.178900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.178864 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.179172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.179154 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.201756 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.201716 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.201756 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.201752 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfg74\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-kube-api-access-cfg74\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.241989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.241954 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:50.576100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.576066 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh"] Oct 30 02:06:50.576868 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:06:50.576841 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1c0f8a_ab2a_43dc_bda9_e8461b76facc.slice/crio-f3910eeed8f57266fd973f8e2afaf6e5c73aab69d062c97a01d0ee280c835414 WatchSource:0}: Error finding container f3910eeed8f57266fd973f8e2afaf6e5c73aab69d062c97a01d0ee280c835414: Status 404 returned error can't find the container with id f3910eeed8f57266fd973f8e2afaf6e5c73aab69d062c97a01d0ee280c835414 Oct 30 02:06:50.579623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.579573 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:06:50.579758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.579677 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:06:50.579758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:50.579749 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:06:51.264628 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:51.264585 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" event={"ID":"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc","Type":"ContainerStarted","Data":"c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c"} Oct 30 02:06:51.265146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:51.264647 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" event={"ID":"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc","Type":"ContainerStarted","Data":"f3910eeed8f57266fd973f8e2afaf6e5c73aab69d062c97a01d0ee280c835414"} Oct 30 02:06:51.345470 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:51.345401 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podStartSLOduration=2.3453811 podStartE2EDuration="2.3453811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:51.328704357 +0000 UTC m=+2289.831636575" watchObservedRunningTime="2026-10-30 02:06:51.3453811 +0000 UTC m=+2289.848313320" Oct 30 02:06:52.242960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:52.242920 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:06:52.244753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:52.244722 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:06:52.244931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:52.244770 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 02:06:53.243012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:53.242910 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:06:53.243012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:53.242989 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 02:06:54.243003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:54.242957 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:06:54.243356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:54.243328 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 02:06:55.261050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:55.250184 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:06:55.261050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:55.250289 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 02:06:56.242802 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:56.242761 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:06:56.243025 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:56.242818 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 02:06:57.243054 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:57.243002 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 02:06:57.243662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:57.243065 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 02:06:58.222301 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.222258 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn"] Oct 30 02:06:58.227180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.227147 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:06:58.247237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.247196 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-4153e5\"" Oct 30 02:06:58.264974 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.264920 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-4153e5-dockercfg-x8lxw\"" Oct 30 02:06:58.278179 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.271469 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn"] Oct 30 02:06:58.365555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.358804 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9850ac-3004-42da-8ea5-d005e5f5bbca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn\" (UID: \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:06:58.365555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.359094 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpqw\" (UniqueName: \"kubernetes.io/projected/6c9850ac-3004-42da-8ea5-d005e5f5bbca-kube-api-access-8rpqw\") pod \"payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn\" (UID: \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:06:58.415552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.412321 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r"] Oct 30 02:06:58.460592 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.460462 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpqw\" (UniqueName: \"kubernetes.io/projected/6c9850ac-3004-42da-8ea5-d005e5f5bbca-kube-api-access-8rpqw\") pod \"payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn\" (UID: \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:06:58.460858 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.460725 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9850ac-3004-42da-8ea5-d005e5f5bbca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn\" (UID: \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:06:58.467421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.465798 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:06:58.469099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.469031 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9850ac-3004-42da-8ea5-d005e5f5bbca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn\" (UID: \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:06:58.495700 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.494988 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r"] Oct 30 02:06:58.495700 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.495632 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpqw\" (UniqueName: \"kubernetes.io/projected/6c9850ac-3004-42da-8ea5-d005e5f5bbca-kube-api-access-8rpqw\") pod \"payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn\" (UID: \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:06:58.544779 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.544718 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:06:58.562209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.562162 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6267677-8af0-44d5-9d95-8e8eb629463e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r\" (UID: \"b6267677-8af0-44d5-9d95-8e8eb629463e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:06:58.562517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.562475 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w82m\" (UniqueName: \"kubernetes.io/projected/b6267677-8af0-44d5-9d95-8e8eb629463e-kube-api-access-9w82m\") pod \"payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r\" (UID: \"b6267677-8af0-44d5-9d95-8e8eb629463e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:06:58.664029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.663978 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6267677-8af0-44d5-9d95-8e8eb629463e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r\" (UID: \"b6267677-8af0-44d5-9d95-8e8eb629463e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:06:58.664366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.664181 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w82m\" (UniqueName: \"kubernetes.io/projected/b6267677-8af0-44d5-9d95-8e8eb629463e-kube-api-access-9w82m\") pod \"payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r\" (UID: \"b6267677-8af0-44d5-9d95-8e8eb629463e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:06:58.665130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.665100 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6267677-8af0-44d5-9d95-8e8eb629463e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r\" (UID: \"b6267677-8af0-44d5-9d95-8e8eb629463e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:06:58.694243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.694182 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w82m\" (UniqueName: \"kubernetes.io/projected/b6267677-8af0-44d5-9d95-8e8eb629463e-kube-api-access-9w82m\") pod \"payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r\" (UID: \"b6267677-8af0-44d5-9d95-8e8eb629463e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:06:58.801077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.800967 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:06:58.874478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:58.870696 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn"] Oct 30 02:06:58.878643 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:06:58.878604 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9850ac_3004_42da_8ea5_d005e5f5bbca.slice/crio-8a3ff9c5f210ad7f700ac892e29ff2867573437a8f1b671475c366eb3f78048b WatchSource:0}: Error finding container 8a3ff9c5f210ad7f700ac892e29ff2867573437a8f1b671475c366eb3f78048b: Status 404 returned error can't find the container with id 8a3ff9c5f210ad7f700ac892e29ff2867573437a8f1b671475c366eb3f78048b Oct 30 02:06:59.131576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.131501 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r"] Oct 30 02:06:59.132109 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:06:59.132079 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6267677_8af0_44d5_9d95_8e8eb629463e.slice/crio-87d62d66af537d3d072b0ce2710d8eb4d2bf2d6aaf9227e2f4be456188d7c43e WatchSource:0}: Error finding container 87d62d66af537d3d072b0ce2710d8eb4d2bf2d6aaf9227e2f4be456188d7c43e: Status 404 returned error can't find the container with id 87d62d66af537d3d072b0ce2710d8eb4d2bf2d6aaf9227e2f4be456188d7c43e Oct 30 02:06:59.243215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.243172 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:06:59.243427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.243237 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:59.417722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.417600 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" event={"ID":"b6267677-8af0-44d5-9d95-8e8eb629463e","Type":"ContainerStarted","Data":"5bc0bad9a453c4a2ac671e105a4c8073503ff578b66ebec6a26086c49648e3c6"} Oct 30 02:06:59.417722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.417677 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" event={"ID":"b6267677-8af0-44d5-9d95-8e8eb629463e","Type":"ContainerStarted","Data":"87d62d66af537d3d072b0ce2710d8eb4d2bf2d6aaf9227e2f4be456188d7c43e"} Oct 30 02:06:59.417722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.417709 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:06:59.419513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.419483 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" event={"ID":"6c9850ac-3004-42da-8ea5-d005e5f5bbca","Type":"ContainerStarted","Data":"6d41cf34e8f4c2974bb18ffe21197f8d96bc27ecc4104e440f26f4edee183c26"} Oct 30 02:06:59.419513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.419545 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" event={"ID":"6c9850ac-3004-42da-8ea5-d005e5f5bbca","Type":"ContainerStarted","Data":"8a3ff9c5f210ad7f700ac892e29ff2867573437a8f1b671475c366eb3f78048b"} Oct 30 02:06:59.419818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.419714 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:06:59.463264 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.463203 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" podStartSLOduration=1.463186425 podStartE2EDuration="1.463186425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:59.46228283 +0000 UTC m=+2297.965215073" watchObservedRunningTime="2026-10-30 02:06:59.463186425 +0000 UTC m=+2297.966118647" Oct 30 02:06:59.507332 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:06:59.507266 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" podStartSLOduration=1.507233689 podStartE2EDuration="1.507233689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:59.505846043 +0000 UTC m=+2298.008778289" watchObservedRunningTime="2026-10-30 02:06:59.507233689 +0000 UTC m=+2298.010165909" Oct 30 02:07:00.281066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:00.281009 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:07:00.281282 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:00.281092 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:07:00.363863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:00.363804 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:07:00.365304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:00.365272 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:07:00.367324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:00.367295 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:07:01.060064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.060019 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595"] Oct 30 02:07:01.064863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.064837 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.085946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.085904 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b4a561-serving-cert\"" Oct 30 02:07:01.092693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.092639 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595"] Oct 30 02:07:01.170736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.170699 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-metrics-certs\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.170946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.170830 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.170946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.170887 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0690fb5d-2223-4310-a044-3f179c073ede-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.171060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.171047 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9msn\" (UniqueName: \"kubernetes.io/projected/0690fb5d-2223-4310-a044-3f179c073ede-kube-api-access-g9msn\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.272273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.272220 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0690fb5d-2223-4310-a044-3f179c073ede-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.272471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.272380 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9msn\" (UniqueName: \"kubernetes.io/projected/0690fb5d-2223-4310-a044-3f179c073ede-kube-api-access-g9msn\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.272471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.272443 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-metrics-certs\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.272567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.272497 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.273044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.273014 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0690fb5d-2223-4310-a044-3f179c073ede-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.275227 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.275206 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.275301 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.275222 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-metrics-certs\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.331519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.331435 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9msn\" (UniqueName: \"kubernetes.io/projected/0690fb5d-2223-4310-a044-3f179c073ede-kube-api-access-g9msn\") pod \"maas-api-e2e-praxis-b-b4a561-78675997b9-s2595\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.384026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.383986 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:01.717558 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:07:01.717500 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0690fb5d_2223_4310_a044_3f179c073ede.slice/crio-3c4b105ddbb47bb2cd6c9529a11399a75efd8bea95db2b122ac13c029b3f4bae WatchSource:0}: Error finding container 3c4b105ddbb47bb2cd6c9529a11399a75efd8bea95db2b122ac13c029b3f4bae: Status 404 returned error can't find the container with id 3c4b105ddbb47bb2cd6c9529a11399a75efd8bea95db2b122ac13c029b3f4bae Oct 30 02:07:01.719366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.719334 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595"] Oct 30 02:07:01.719685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:01.719665 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:07:02.455272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:02.455197 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" event={"ID":"0690fb5d-2223-4310-a044-3f179c073ede","Type":"ContainerStarted","Data":"f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5"} Oct 30 02:07:02.455798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:02.455303 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" event={"ID":"0690fb5d-2223-4310-a044-3f179c073ede","Type":"ContainerStarted","Data":"3c4b105ddbb47bb2cd6c9529a11399a75efd8bea95db2b122ac13c029b3f4bae"} Oct 30 02:07:02.455798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:02.455336 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:02.501665 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:02.501584 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" podStartSLOduration=1.15855983 podStartE2EDuration="1.501519777s" totalImagesPullingTime="342.959947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:02.50118049 +0000 UTC m=+2301.004112710" watchObservedRunningTime="2026-10-30 02:07:02.501519777 +0000 UTC m=+2301.004452040" Oct 30 02:07:05.007391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.007352 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj"] Oct 30 02:07:05.012420 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.012386 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.033730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.033689 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj"] Oct 30 02:07:05.083316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.083271 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19141fbc-b473-46d4-8383-658fdc21da18-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.083512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.083421 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.083602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.083509 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.083602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.083594 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19141fbc-b473-46d4-8383-658fdc21da18-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.083675 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.083620 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.083675 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.083643 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.083802 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.083782 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.083837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.083814 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.083956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.083943 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2t7\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-kube-api-access-hp2t7\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.185567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.185481 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2t7\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-kube-api-access-hp2t7\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.185567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.185574 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19141fbc-b473-46d4-8383-658fdc21da18-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.185878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.185649 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.185878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.185688 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.185878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.185722 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19141fbc-b473-46d4-8383-658fdc21da18-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.185878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.185759 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.185878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.185784 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.185878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.185856 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.186221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.185887 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.186221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.186164 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.186356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.186211 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.186356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.186300 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.186356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.186316 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.186513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.186458 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19141fbc-b473-46d4-8383-658fdc21da18-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.188289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.188266 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.188416 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.188311 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19141fbc-b473-46d4-8383-658fdc21da18-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.211010 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.210977 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.211178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.211016 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2t7\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-kube-api-access-hp2t7\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.336873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.336767 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:05.666741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.666706 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj"] Oct 30 02:07:05.668072 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:07:05.668028 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19141fbc_b473_46d4_8383_658fdc21da18.slice/crio-d80ef24bf8b6b31fe8ec6feb59fc252d36122292e355d10483bbb46fafd121e7 WatchSource:0}: Error finding container d80ef24bf8b6b31fe8ec6feb59fc252d36122292e355d10483bbb46fafd121e7: Status 404 returned error can't find the container with id d80ef24bf8b6b31fe8ec6feb59fc252d36122292e355d10483bbb46fafd121e7 Oct 30 02:07:05.671247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.671208 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:07:05.671362 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.671303 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:07:05.671362 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:05.671348 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:07:06.497216 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:06.497160 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" event={"ID":"19141fbc-b473-46d4-8383-658fdc21da18","Type":"ContainerStarted","Data":"1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802"} Oct 30 02:07:06.497727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:06.497240 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" event={"ID":"19141fbc-b473-46d4-8383-658fdc21da18","Type":"ContainerStarted","Data":"d80ef24bf8b6b31fe8ec6feb59fc252d36122292e355d10483bbb46fafd121e7"} Oct 30 02:07:06.561539 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:06.561467 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" podStartSLOduration=2.561451516 podStartE2EDuration="2.561451516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:06.559469457 +0000 UTC m=+2305.062401711" watchObservedRunningTime="2026-10-30 02:07:06.561451516 +0000 UTC m=+2305.064383809" Oct 30 02:07:07.337217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:07.337177 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:07.339426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:07.339392 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:07:07.339672 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:07.339470 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 02:07:08.337936 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:08.337892 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:07:08.338357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:08.337952 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 02:07:08.493110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:08.493072 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:07:09.337986 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:09.337947 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:07:09.338427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:09.338007 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 02:07:10.338175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:10.338138 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:07:10.338632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:10.338210 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 02:07:10.431469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:10.431434 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:07:10.431703 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:10.431509 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:07:11.337767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:11.337721 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:07:11.337988 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:11.337792 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 02:07:12.337374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:12.337322 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 02:07:12.337873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:12.337377 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 02:07:13.342989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:13.342952 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:13.343470 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:13.343447 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:13.344205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:13.344186 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:07:17.630285 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:07:17.630240 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:07:21.218335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:21.218282 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 30 02:07:27.689468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.689430 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9"] Oct 30 02:07:27.694037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.694012 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:27.722370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.722332 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-24fb97\"" Oct 30 02:07:27.740439 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.740397 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9"] Oct 30 02:07:27.758153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.758111 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp"] Oct 30 02:07:27.763062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.763022 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:27.779906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.779866 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-24fb97-dockercfg-6tvgm\"" Oct 30 02:07:27.785018 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.784986 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp"] Oct 30 02:07:27.864423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.862820 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp\" (UID: \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:27.864423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.862989 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbq9\" (UniqueName: \"kubernetes.io/projected/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-kube-api-access-7qbq9\") pod \"payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp\" (UID: \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:27.864423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.863053 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55bhk\" (UniqueName: \"kubernetes.io/projected/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-kube-api-access-55bhk\") pod \"payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9\" (UID: \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:27.864423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.863101 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9\" (UID: \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:27.967012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.965971 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp\" (UID: \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:27.967012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.966103 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbq9\" (UniqueName: \"kubernetes.io/projected/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-kube-api-access-7qbq9\") pod \"payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp\" (UID: \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:27.967012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.966148 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55bhk\" (UniqueName: \"kubernetes.io/projected/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-kube-api-access-55bhk\") pod \"payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9\" (UID: \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:27.967012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.966192 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9\" (UID: \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:27.972199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.967512 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9\" (UID: \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:27.972199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.968370 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp\" (UID: \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:27.997312 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:27.997279 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbq9\" (UniqueName: \"kubernetes.io/projected/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-kube-api-access-7qbq9\") pod \"payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp\" (UID: \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:28.004917 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:28.004878 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bhk\" (UniqueName: \"kubernetes.io/projected/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-kube-api-access-55bhk\") pod \"payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9\" (UID: \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:28.010213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:28.010174 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:28.086077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:28.086052 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:28.332322 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:07:28.332286 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a23052_76c4_4b5f_9e92_95aeb0d322e7.slice/crio-cd6036ca7a0a53fa718f59058235854c583f7fd2ee89cd482f93ac6aa98d1582 WatchSource:0}: Error finding container cd6036ca7a0a53fa718f59058235854c583f7fd2ee89cd482f93ac6aa98d1582: Status 404 returned error can't find the container with id cd6036ca7a0a53fa718f59058235854c583f7fd2ee89cd482f93ac6aa98d1582 Oct 30 02:07:28.332462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:28.332319 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9"] Oct 30 02:07:28.432023 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:07:28.431926 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d4aa6e_66fa_4c8f_b1d0_916477867b42.slice/crio-9b9f913444231e8eb94d670424400095b1d7aad42cdf3aecce1559ee2ddd7cc4 WatchSource:0}: Error finding container 9b9f913444231e8eb94d670424400095b1d7aad42cdf3aecce1559ee2ddd7cc4: Status 404 returned error can't find the container with id 9b9f913444231e8eb94d670424400095b1d7aad42cdf3aecce1559ee2ddd7cc4 Oct 30 02:07:28.432203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:28.432175 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp"] Oct 30 02:07:28.850319 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:28.850267 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2"] Oct 30 02:07:28.850825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:28.850596 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" containerID="cri-o://1596114c23219865b5e6792b97e49f2b40f1cfbb160514854e28aeb355c99868" gracePeriod=30 Oct 30 02:07:29.012789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:29.012749 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" event={"ID":"d6d4aa6e-66fa-4c8f-b1d0-916477867b42","Type":"ContainerStarted","Data":"fbd0e71f50b7b48f072a2e41adb03cc30e325c6de0d1e230e734392ca4ac8850"} Oct 30 02:07:29.012975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:29.012822 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" event={"ID":"d6d4aa6e-66fa-4c8f-b1d0-916477867b42","Type":"ContainerStarted","Data":"9b9f913444231e8eb94d670424400095b1d7aad42cdf3aecce1559ee2ddd7cc4"} Oct 30 02:07:29.012975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:29.012849 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:29.016154 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:29.016120 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" event={"ID":"c5a23052-76c4-4b5f-9e92-95aeb0d322e7","Type":"ContainerStarted","Data":"d8333b16222fadb547455db18b1b76e0f0808a922bb88c1736b9f75ade3522dd"} Oct 30 02:07:29.016317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:29.016173 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" event={"ID":"c5a23052-76c4-4b5f-9e92-95aeb0d322e7","Type":"ContainerStarted","Data":"cd6036ca7a0a53fa718f59058235854c583f7fd2ee89cd482f93ac6aa98d1582"} Oct 30 02:07:29.016915 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:29.016897 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:29.060758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:29.060699 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" podStartSLOduration=2.06068128 podStartE2EDuration="2.06068128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:29.056590099 +0000 UTC m=+2327.559522319" watchObservedRunningTime="2026-10-30 02:07:29.06068128 +0000 UTC m=+2327.563613500" Oct 30 02:07:29.103768 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:29.103653 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" podStartSLOduration=2.103631633 podStartE2EDuration="2.103631633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:29.10026553 +0000 UTC m=+2327.603197750" watchObservedRunningTime="2026-10-30 02:07:29.103631633 +0000 UTC m=+2327.606563857" Oct 30 02:07:31.136424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:31.136381 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j"] Oct 30 02:07:32.040995 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:32.040960 5035 generic.go:356] "Generic (PLEG): container finished" podID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerID="a395e1a73e2180088d7146685523ee09520eb874b6ae2526a22050dd337cce6b" exitCode=0 Oct 30 02:07:32.041197 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:32.041025 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" event={"ID":"ef3cea51-d79d-4f51-9ebb-bd28f0335889","Type":"ContainerDied","Data":"a395e1a73e2180088d7146685523ee09520eb874b6ae2526a22050dd337cce6b"} Oct 30 02:07:32.041515 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:32.041488 5035 scope.go:122] "RemoveContainer" containerID="a395e1a73e2180088d7146685523ee09520eb874b6ae2526a22050dd337cce6b" Oct 30 02:07:32.043197 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:32.043162 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:07:32.043295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:32.043241 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:07:32.043295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:32.043278 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:07:33.050986 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:33.050946 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" event={"ID":"ef3cea51-d79d-4f51-9ebb-bd28f0335889","Type":"ContainerStarted","Data":"3c3f68a72353b32ca0c3f598be510ef3f8790f79c73418d439b4d053cac44ad3"} Oct 30 02:07:33.796888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:33.796840 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:07:33.797112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:33.797085 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:07:33.797190 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:33.797150 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 02:07:34.063311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.063218 5035 generic.go:356] "Generic (PLEG): container finished" podID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerID="1596114c23219865b5e6792b97e49f2b40f1cfbb160514854e28aeb355c99868" exitCode=0 Oct 30 02:07:34.063709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.063398 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" event={"ID":"587cfc66-c33f-40b7-b2a0-19639bfa8aef","Type":"ContainerDied","Data":"1596114c23219865b5e6792b97e49f2b40f1cfbb160514854e28aeb355c99868"} Oct 30 02:07:34.304438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.304072 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 02:07:34.414166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.413252 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-envoy\") pod \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " Oct 30 02:07:34.414166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.413357 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-socket\") pod \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " Oct 30 02:07:34.414166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.413404 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-token\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-token\") pod \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " Oct 30 02:07:34.414166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.413579 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-data\") pod \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " Oct 30 02:07:34.414166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.413627 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-credential-socket\") pod \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " Oct 30 02:07:34.414166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.413668 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istiod-ca-cert\") pod \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " Oct 30 02:07:34.414166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.413720 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-podinfo\") pod \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " Oct 30 02:07:34.414166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.413784 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-certs\") pod \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " Oct 30 02:07:34.414166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.413822 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-kube-api-access-qblrj\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-kube-api-access-qblrj\") pod \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\" (UID: \"587cfc66-c33f-40b7-b2a0-19639bfa8aef\") " Oct 30 02:07:34.414832 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.414352 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istiod-ca-cert" pod "587cfc66-c33f-40b7-b2a0-19639bfa8aef" (UID: "587cfc66-c33f-40b7-b2a0-19639bfa8aef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:34.414832 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.414515 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.414832 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.414768 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-data" pod "587cfc66-c33f-40b7-b2a0-19639bfa8aef" (UID: "587cfc66-c33f-40b7-b2a0-19639bfa8aef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:34.414993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.414950 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-credential-socket" pod "587cfc66-c33f-40b7-b2a0-19639bfa8aef" (UID: "587cfc66-c33f-40b7-b2a0-19639bfa8aef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:34.415694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.415636 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-socket" pod "587cfc66-c33f-40b7-b2a0-19639bfa8aef" (UID: "587cfc66-c33f-40b7-b2a0-19639bfa8aef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:34.416084 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.416060 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-certs" pod "587cfc66-c33f-40b7-b2a0-19639bfa8aef" (UID: "587cfc66-c33f-40b7-b2a0-19639bfa8aef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:34.416801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.416765 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-token" pod "587cfc66-c33f-40b7-b2a0-19639bfa8aef" (UID: "587cfc66-c33f-40b7-b2a0-19639bfa8aef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:34.417841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.417797 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-kube-api-access-qblrj" pod "587cfc66-c33f-40b7-b2a0-19639bfa8aef" (UID: "587cfc66-c33f-40b7-b2a0-19639bfa8aef"). InnerVolumeSpecName "kube-api-access-qblrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:34.418505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.418466 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-podinfo" pod "587cfc66-c33f-40b7-b2a0-19639bfa8aef" (UID: "587cfc66-c33f-40b7-b2a0-19639bfa8aef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:07:34.419996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.419958 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-envoy" pod "587cfc66-c33f-40b7-b2a0-19639bfa8aef" (UID: "587cfc66-c33f-40b7-b2a0-19639bfa8aef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:34.515945 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.515907 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.515945 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.515941 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.515945 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.515954 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.516238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.515963 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.516238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.515972 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.516238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.515981 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587cfc66-c33f-40b7-b2a0-19639bfa8aef-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.516238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.515990 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587cfc66-c33f-40b7-b2a0-19639bfa8aef-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.516238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.516000 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qblrj\" (UniqueName: \"kubernetes.io/projected/587cfc66-c33f-40b7-b2a0-19639bfa8aef-kube-api-access-qblrj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.796637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.796588 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:07:34.796843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:34.796668 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 02:07:35.096249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.096156 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" event={"ID":"587cfc66-c33f-40b7-b2a0-19639bfa8aef","Type":"ContainerDied","Data":"e30182a74f686a9ba253ddbdd15ec5e49ec02214f5ff21352a8114d30a8aa9bd"} Oct 30 02:07:35.096249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.096203 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" Oct 30 02:07:35.096800 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.096236 5035 scope.go:122] "RemoveContainer" containerID="1596114c23219865b5e6792b97e49f2b40f1cfbb160514854e28aeb355c99868" Oct 30 02:07:35.107542 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.107499 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst"] Oct 30 02:07:35.215904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.215844 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2"] Oct 30 02:07:35.220009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.219966 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2"] Oct 30 02:07:35.307562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.289629 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gjlm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:07:35.307562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.289711 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gjlm2" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:07:35.644614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.644577 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" path="/var/lib/kubelet/pods/587cfc66-c33f-40b7-b2a0-19639bfa8aef/volumes" Oct 30 02:07:35.802551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.800948 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:07:35.802551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:35.801018 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 02:07:36.137486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:36.137437 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" podUID="e59d5a6e-c78e-4d7b-8edb-8b014303aad7" containerName="payload-processing" containerID="cri-o://f568f2d2244455df186aee27c2d84ea6a5bc9891841dd9707334faf7d1d8b2db" gracePeriod=25 Oct 30 02:07:36.797157 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:36.797111 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:07:36.797359 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:36.797191 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 02:07:37.797074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:37.797031 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 02:07:37.797512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:37.797087 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 02:07:38.723171 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:38.723123 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" podUID="e59d5a6e-c78e-4d7b-8edb-8b014303aad7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:07:38.802114 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:38.802064 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:07:38.802579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:38.802443 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:07:38.803254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:38.803235 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:07:40.027026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:40.026990 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:07:40.108397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:40.108319 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" podUID="42ee0a8c-6993-4316-94c4-fa91b3db48db" containerName="payload-processing" containerID="cri-o://e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87" gracePeriod=25 Oct 30 02:07:40.390682 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:40.390652 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:07:40.544295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:40.544195 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ee0a8c-6993-4316-94c4-fa91b3db48db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ee0a8c-6993-4316-94c4-fa91b3db48db-plugins-config-volume\") pod \"42ee0a8c-6993-4316-94c4-fa91b3db48db\" (UID: \"42ee0a8c-6993-4316-94c4-fa91b3db48db\") " Oct 30 02:07:40.544464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:40.544426 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ee0a8c-6993-4316-94c4-fa91b3db48db-kube-api-access-nv4kk\" (UniqueName: \"kubernetes.io/projected/42ee0a8c-6993-4316-94c4-fa91b3db48db-kube-api-access-nv4kk\") pod \"42ee0a8c-6993-4316-94c4-fa91b3db48db\" (UID: \"42ee0a8c-6993-4316-94c4-fa91b3db48db\") " Oct 30 02:07:40.544582 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:40.544546 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ee0a8c-6993-4316-94c4-fa91b3db48db-plugins-config-volume" pod "42ee0a8c-6993-4316-94c4-fa91b3db48db" (UID: "42ee0a8c-6993-4316-94c4-fa91b3db48db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:40.544904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:40.544879 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ee0a8c-6993-4316-94c4-fa91b3db48db-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:40.546657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:40.546635 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ee0a8c-6993-4316-94c4-fa91b3db48db-kube-api-access-nv4kk" pod "42ee0a8c-6993-4316-94c4-fa91b3db48db" (UID: "42ee0a8c-6993-4316-94c4-fa91b3db48db"). InnerVolumeSpecName "kube-api-access-nv4kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:40.646349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:40.646294 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv4kk\" (UniqueName: \"kubernetes.io/projected/42ee0a8c-6993-4316-94c4-fa91b3db48db-kube-api-access-nv4kk\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:41.035574 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.035498 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:07:41.145774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.145739 5035 generic.go:356] "Generic (PLEG): container finished" podID="42ee0a8c-6993-4316-94c4-fa91b3db48db" containerID="e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87" exitCode=0 Oct 30 02:07:41.145774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.145793 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" Oct 30 02:07:41.145774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.145807 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" event={"ID":"42ee0a8c-6993-4316-94c4-fa91b3db48db","Type":"ContainerDied","Data":"e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87"} Oct 30 02:07:41.146142 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.145849 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst" event={"ID":"42ee0a8c-6993-4316-94c4-fa91b3db48db","Type":"ContainerDied","Data":"06a334f77a73122a32d0a06b37ba3b53bae2c75190d5c45020d1b77789432231"} Oct 30 02:07:41.146142 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.145872 5035 scope.go:122] "RemoveContainer" containerID="e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87" Oct 30 02:07:41.157370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.157337 5035 scope.go:122] "RemoveContainer" containerID="e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87" Oct 30 02:07:41.157857 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:07:41.157830 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87\": container with ID starting with e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87 not found: ID does not exist" containerID="e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87" Oct 30 02:07:41.157970 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.157867 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87"} err="failed to get container status \"e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87\": rpc error: code = NotFound desc = could not find container \"e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87\": container with ID starting with e1c0356a9d124a8815e7bacbeeec2c1dc562ccb070bcf7d36d79522f5da8ab87 not found: ID does not exist" Oct 30 02:07:41.222398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.222357 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst"] Oct 30 02:07:41.228055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.228009 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-428529-6b49dc7d56-9nrst"] Oct 30 02:07:41.654231 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:41.654197 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ee0a8c-6993-4316-94c4-fa91b3db48db" path="/var/lib/kubelet/pods/42ee0a8c-6993-4316-94c4-fa91b3db48db/volumes" Oct 30 02:07:42.102265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.102222 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2"] Oct 30 02:07:42.102931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.102916 5035 cpu_manager.go:414] "containerMap: removing container" podUID="42ee0a8c-6993-4316-94c4-fa91b3db48db" containerName="payload-processing" Oct 30 02:07:42.102973 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.102932 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ee0a8c-6993-4316-94c4-fa91b3db48db" containerName="payload-processing" Oct 30 02:07:42.103009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.102973 5035 cpu_manager.go:414] "containerMap: removing container" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" Oct 30 02:07:42.103009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.102979 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" Oct 30 02:07:42.103079 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.103069 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ee0a8c-6993-4316-94c4-fa91b3db48db" containerName="payload-processing" Oct 30 02:07:42.103113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.103086 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="587cfc66-c33f-40b7-b2a0-19639bfa8aef" containerName="istio-proxy" Oct 30 02:07:42.108324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.108259 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.136174 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.136141 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2"] Oct 30 02:07:42.169140 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.169091 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-certs\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.169364 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.169157 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcln\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-kube-api-access-7lcln\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.169364 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.169282 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-data\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.169507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.169424 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-socket\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.169659 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.169638 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istiod-ca-cert\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.169751 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.169728 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-envoy\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.169817 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.169797 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-podinfo\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.169905 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.169875 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-token\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.170058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.169957 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-credential-socket\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.170512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.170482 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-17e747-openshift-default-dockercfg-4nll2\"" Oct 30 02:07:42.270885 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.270835 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-data\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.270916 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-socket\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271022 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istiod-ca-cert\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271068 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-envoy\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271110 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-podinfo\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271153 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-token\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271203 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-credential-socket\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271301 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-certs\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271308 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-data\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271333 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcln\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-kube-api-access-7lcln\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271412 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-socket\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271718 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271693 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-credential-socket\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271734 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-certs\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.271818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.271787 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istiod-ca-cert\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.273632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.273603 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-envoy\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.273924 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.273903 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-podinfo\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.305443 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.305381 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-token\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.306825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.306720 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcln\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-kube-api-access-7lcln\") pod \"e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:42.428616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:42.426854 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:43.165033 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:43.164970 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:07:43.165451 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:43.165126 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:07:43.165451 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:43.165210 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:07:43.166693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:43.165831 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2"] Oct 30 02:07:43.251771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:43.251723 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" event={"ID":"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3","Type":"ContainerStarted","Data":"fafa363fdda5472506ae7dfa74fba48a7c863cd5274d3d9d1733577d8162e86a"} Oct 30 02:07:44.262076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:44.262029 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" event={"ID":"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3","Type":"ContainerStarted","Data":"5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5"} Oct 30 02:07:44.341118 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:44.341061 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" podStartSLOduration=2.341043653 podStartE2EDuration="2.341043653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:44.324777139 +0000 UTC m=+2342.827709361" watchObservedRunningTime="2026-10-30 02:07:44.341043653 +0000 UTC m=+2342.843975932" Oct 30 02:07:44.429952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:44.429906 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:44.436209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:44.436170 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:45.269469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:45.269429 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:45.270584 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:45.270559 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:07:47.369878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.369839 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg"] Oct 30 02:07:47.375979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.375949 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.398747 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.398711 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg"] Oct 30 02:07:47.406323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.406288 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-24fb97-serving-cert\"" Oct 30 02:07:47.422548 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.422496 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.422770 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.422626 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nn5c\" (UniqueName: \"kubernetes.io/projected/b81cd9b8-f172-4579-85fe-b447b1187777-kube-api-access-4nn5c\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.423003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.422972 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.423087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.423023 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cd9b8-f172-4579-85fe-b447b1187777-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.524949 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.524904 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.524949 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.524954 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cd9b8-f172-4579-85fe-b447b1187777-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.525240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.525017 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.525240 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:07:47.525043 5035 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-serving-cert: secret "maas-api-e2e-shared-a-w3-24fb97-serving-cert" not found Oct 30 02:07:47.525240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.525072 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nn5c\" (UniqueName: \"kubernetes.io/projected/b81cd9b8-f172-4579-85fe-b447b1187777-kube-api-access-4nn5c\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.525240 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:07:47.525123 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls podName:b81cd9b8-f172-4579-85fe-b447b1187777 nodeName:}" failed. No retries permitted until 2026-10-30 02:07:48.025101238 +0000 UTC m=+2346.528033440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls") pod "maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" (UID: "b81cd9b8-f172-4579-85fe-b447b1187777") : secret "maas-api-e2e-shared-a-w3-24fb97-serving-cert" not found Oct 30 02:07:47.525727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.525699 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cd9b8-f172-4579-85fe-b447b1187777-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.527741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.527714 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.578467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:47.578426 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nn5c\" (UniqueName: \"kubernetes.io/projected/b81cd9b8-f172-4579-85fe-b447b1187777-kube-api-access-4nn5c\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:47.630931 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:07:47.630841 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:07:48.039371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:48.039333 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:48.041954 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:48.041918 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:48.291442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:48.291359 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:48.607788 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:48.607752 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg"] Oct 30 02:07:48.609193 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:07:48.609163 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81cd9b8_f172_4579_85fe_b447b1187777.slice/crio-9c0afc0556842c593791d64222aab174ee3772854b9f0be3a7174d012255101a WatchSource:0}: Error finding container 9c0afc0556842c593791d64222aab174ee3772854b9f0be3a7174d012255101a: Status 404 returned error can't find the container with id 9c0afc0556842c593791d64222aab174ee3772854b9f0be3a7174d012255101a Oct 30 02:07:48.777744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:48.777698 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" podUID="e59d5a6e-c78e-4d7b-8edb-8b014303aad7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:07:49.300542 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:49.300433 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" event={"ID":"b81cd9b8-f172-4579-85fe-b447b1187777","Type":"ContainerStarted","Data":"4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd"} Oct 30 02:07:49.300542 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:49.300491 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" event={"ID":"b81cd9b8-f172-4579-85fe-b447b1187777","Type":"ContainerStarted","Data":"9c0afc0556842c593791d64222aab174ee3772854b9f0be3a7174d012255101a"} Oct 30 02:07:49.300804 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:49.300523 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:49.348078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:49.348016 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" podStartSLOduration=1.9910894350000001 podStartE2EDuration="2.347997085s" totalImagesPullingTime="356.90765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:49.346994056 +0000 UTC m=+2347.849926277" watchObservedRunningTime="2026-10-30 02:07:49.347997085 +0000 UTC m=+2347.850929304" Oct 30 02:07:53.161165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.161124 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm"] Oct 30 02:07:53.161889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.161483 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" podUID="de6ff5cf-766f-49c0-b962-305c86af4280" containerName="maas-api" containerID="cri-o://3e5a3d326ec9a4679d8a29ec621a8d895bc279190804ccb4996c7c687e4a9240" gracePeriod=30 Oct 30 02:07:53.335657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.335603 5035 generic.go:356] "Generic (PLEG): container finished" podID="de6ff5cf-766f-49c0-b962-305c86af4280" containerID="3e5a3d326ec9a4679d8a29ec621a8d895bc279190804ccb4996c7c687e4a9240" exitCode=0 Oct 30 02:07:53.335886 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.335684 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" event={"ID":"de6ff5cf-766f-49c0-b962-305c86af4280","Type":"ContainerDied","Data":"3e5a3d326ec9a4679d8a29ec621a8d895bc279190804ccb4996c7c687e4a9240"} Oct 30 02:07:53.473882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.473851 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:07:53.567028 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.566986 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-metrics-certs\") pod \"de6ff5cf-766f-49c0-b962-305c86af4280\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " Oct 30 02:07:53.567252 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.567144 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6ff5cf-766f-49c0-b962-305c86af4280-kube-api-access-b6qwt\" (UniqueName: \"kubernetes.io/projected/de6ff5cf-766f-49c0-b962-305c86af4280-kube-api-access-b6qwt\") pod \"de6ff5cf-766f-49c0-b962-305c86af4280\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " Oct 30 02:07:53.567252 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.567226 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de6ff5cf-766f-49c0-b962-305c86af4280-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6ff5cf-766f-49c0-b962-305c86af4280-openshift-service-ca\") pod \"de6ff5cf-766f-49c0-b962-305c86af4280\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " Oct 30 02:07:53.567374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.567259 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls\") pod \"de6ff5cf-766f-49c0-b962-305c86af4280\" (UID: \"de6ff5cf-766f-49c0-b962-305c86af4280\") " Oct 30 02:07:53.576176 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.576134 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6ff5cf-766f-49c0-b962-305c86af4280-openshift-service-ca" pod "de6ff5cf-766f-49c0-b962-305c86af4280" (UID: "de6ff5cf-766f-49c0-b962-305c86af4280"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:53.640549 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.635842 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6ff5cf-766f-49c0-b962-305c86af4280-kube-api-access-b6qwt" pod "de6ff5cf-766f-49c0-b962-305c86af4280" (UID: "de6ff5cf-766f-49c0-b962-305c86af4280"). InnerVolumeSpecName "kube-api-access-b6qwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:53.677558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.672116 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6qwt\" (UniqueName: \"kubernetes.io/projected/de6ff5cf-766f-49c0-b962-305c86af4280-kube-api-access-b6qwt\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:53.677558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.672165 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6ff5cf-766f-49c0-b962-305c86af4280-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:53.704910 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.702352 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls" pod "de6ff5cf-766f-49c0-b962-305c86af4280" (UID: "de6ff5cf-766f-49c0-b962-305c86af4280"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:07:53.721618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.706039 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-metrics-certs" pod "de6ff5cf-766f-49c0-b962-305c86af4280" (UID: "de6ff5cf-766f-49c0-b962-305c86af4280"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:07:53.784374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.784330 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:53.784374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:53.784371 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6ff5cf-766f-49c0-b962-305c86af4280-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:54.378670 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:54.378385 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" event={"ID":"de6ff5cf-766f-49c0-b962-305c86af4280","Type":"ContainerDied","Data":"13deb363a0832853903c3f280726afceadbbc805555e7747f84282843779be5d"} Oct 30 02:07:54.378670 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:54.378456 5035 scope.go:122] "RemoveContainer" containerID="3e5a3d326ec9a4679d8a29ec621a8d895bc279190804ccb4996c7c687e4a9240" Oct 30 02:07:54.378670 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:54.378397 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm" Oct 30 02:07:54.440726 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:54.440615 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm"] Oct 30 02:07:54.453472 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:54.453435 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2979be-7b8dcd75cf-nlmgm"] Oct 30 02:07:55.313073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:55.313043 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:07:55.645881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:55.645791 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de6ff5cf-766f-49c0-b962-305c86af4280" path="/var/lib/kubelet/pods/de6ff5cf-766f-49c0-b962-305c86af4280/volumes" Oct 30 02:07:56.398734 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.398633 5035 generic.go:356] "Generic (PLEG): container finished" podID="e59d5a6e-c78e-4d7b-8edb-8b014303aad7" containerID="f568f2d2244455df186aee27c2d84ea6a5bc9891841dd9707334faf7d1d8b2db" exitCode=0 Oct 30 02:07:56.398734 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.398715 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" event={"ID":"e59d5a6e-c78e-4d7b-8edb-8b014303aad7","Type":"ContainerDied","Data":"f568f2d2244455df186aee27c2d84ea6a5bc9891841dd9707334faf7d1d8b2db"} Oct 30 02:07:56.440603 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.440571 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:07:56.544939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.544891 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-plugins-config-volume\") pod \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\" (UID: \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\") " Oct 30 02:07:56.545124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.544972 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-kube-api-access-9gg77\" (UniqueName: \"kubernetes.io/projected/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-kube-api-access-9gg77\") pod \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\" (UID: \"e59d5a6e-c78e-4d7b-8edb-8b014303aad7\") " Oct 30 02:07:56.545366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.545338 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-plugins-config-volume" pod "e59d5a6e-c78e-4d7b-8edb-8b014303aad7" (UID: "e59d5a6e-c78e-4d7b-8edb-8b014303aad7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:56.545757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.545735 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:56.547194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.547159 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-kube-api-access-9gg77" pod "e59d5a6e-c78e-4d7b-8edb-8b014303aad7" (UID: "e59d5a6e-c78e-4d7b-8edb-8b014303aad7"). InnerVolumeSpecName "kube-api-access-9gg77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:56.646740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.646697 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gg77\" (UniqueName: \"kubernetes.io/projected/e59d5a6e-c78e-4d7b-8edb-8b014303aad7-kube-api-access-9gg77\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:07:56.934915 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.934871 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw"] Oct 30 02:07:56.935592 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.935575 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e59d5a6e-c78e-4d7b-8edb-8b014303aad7" containerName="payload-processing" Oct 30 02:07:56.935658 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.935592 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59d5a6e-c78e-4d7b-8edb-8b014303aad7" containerName="payload-processing" Oct 30 02:07:56.935658 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.935627 5035 cpu_manager.go:414] "containerMap: removing container" podUID="de6ff5cf-766f-49c0-b962-305c86af4280" containerName="maas-api" Oct 30 02:07:56.935658 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.935632 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6ff5cf-766f-49c0-b962-305c86af4280" containerName="maas-api" Oct 30 02:07:56.935778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.935768 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6ff5cf-766f-49c0-b962-305c86af4280" containerName="maas-api" Oct 30 02:07:56.935821 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.935786 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59d5a6e-c78e-4d7b-8edb-8b014303aad7" containerName="payload-processing" Oct 30 02:07:56.939494 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.939471 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:07:56.956092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.956043 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-9a1cd0\"" Oct 30 02:07:56.956371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.956057 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-9a1cd0-dockercfg-jwq2m\"" Oct 30 02:07:56.961109 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:56.961072 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw"] Oct 30 02:07:57.070562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.055571 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bab1e7-a639-4904-a00f-05e71080b13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw\" (UID: \"a6bab1e7-a639-4904-a00f-05e71080b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:07:57.070562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.055669 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92kbt\" (UniqueName: \"kubernetes.io/projected/a6bab1e7-a639-4904-a00f-05e71080b13e-kube-api-access-92kbt\") pod \"payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw\" (UID: \"a6bab1e7-a639-4904-a00f-05e71080b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:07:57.167563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.162621 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bab1e7-a639-4904-a00f-05e71080b13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw\" (UID: \"a6bab1e7-a639-4904-a00f-05e71080b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:07:57.168008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.167972 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92kbt\" (UniqueName: \"kubernetes.io/projected/a6bab1e7-a639-4904-a00f-05e71080b13e-kube-api-access-92kbt\") pod \"payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw\" (UID: \"a6bab1e7-a639-4904-a00f-05e71080b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:07:57.173604 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.169511 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bab1e7-a639-4904-a00f-05e71080b13e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw\" (UID: \"a6bab1e7-a639-4904-a00f-05e71080b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:07:57.195212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.195137 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92kbt\" (UniqueName: \"kubernetes.io/projected/a6bab1e7-a639-4904-a00f-05e71080b13e-kube-api-access-92kbt\") pod \"payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw\" (UID: \"a6bab1e7-a639-4904-a00f-05e71080b13e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:07:57.255426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.255391 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:07:57.410118 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.410075 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" Oct 30 02:07:57.410375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.410087 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j" event={"ID":"e59d5a6e-c78e-4d7b-8edb-8b014303aad7","Type":"ContainerDied","Data":"8345f5cd1919347ced793fef574004821c7795c2fbdd4db645b81c17f308b6e6"} Oct 30 02:07:57.410375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.410245 5035 scope.go:122] "RemoveContainer" containerID="f568f2d2244455df186aee27c2d84ea6a5bc9891841dd9707334faf7d1d8b2db" Oct 30 02:07:57.469228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.469194 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j"] Oct 30 02:07:57.473289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.473256 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2979be-6c69bc79fb-89z4j"] Oct 30 02:07:57.570233 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.570199 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw"] Oct 30 02:07:57.571417 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:07:57.571379 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bab1e7_a639_4904_a00f_05e71080b13e.slice/crio-0a230a352c9fd6af3e049fdf3ce44921057144a7e91d891436497b89e5227b24 WatchSource:0}: Error finding container 0a230a352c9fd6af3e049fdf3ce44921057144a7e91d891436497b89e5227b24: Status 404 returned error can't find the container with id 0a230a352c9fd6af3e049fdf3ce44921057144a7e91d891436497b89e5227b24 Oct 30 02:07:57.644591 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:57.644557 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59d5a6e-c78e-4d7b-8edb-8b014303aad7" path="/var/lib/kubelet/pods/e59d5a6e-c78e-4d7b-8edb-8b014303aad7/volumes" Oct 30 02:07:58.419683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:58.419645 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" event={"ID":"a6bab1e7-a639-4904-a00f-05e71080b13e","Type":"ContainerStarted","Data":"feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664"} Oct 30 02:07:58.420247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:58.419707 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" event={"ID":"a6bab1e7-a639-4904-a00f-05e71080b13e","Type":"ContainerStarted","Data":"0a230a352c9fd6af3e049fdf3ce44921057144a7e91d891436497b89e5227b24"} Oct 30 02:07:58.420247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:58.419738 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:07:58.467666 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:07:58.467607 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" podStartSLOduration=2.467590676 podStartE2EDuration="2.467590676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:58.46346367 +0000 UTC m=+2356.966395889" watchObservedRunningTime="2026-10-30 02:07:58.467590676 +0000 UTC m=+2356.970522900" Oct 30 02:08:07.560908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.560868 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n"] Oct 30 02:08:07.565484 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.565455 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.589820 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.589788 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-17e747-serving-cert\"" Oct 30 02:08:07.596858 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.596818 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n"] Oct 30 02:08:07.680758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.680715 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.680962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.680830 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s96f7\" (UniqueName: \"kubernetes.io/projected/e63eff88-dd24-4f72-ac4b-76f163edd204-kube-api-access-s96f7\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.681078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.681000 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63eff88-dd24-4f72-ac4b-76f163edd204-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.681148 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.681121 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.813607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.811593 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.813607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.811670 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s96f7\" (UniqueName: \"kubernetes.io/projected/e63eff88-dd24-4f72-ac4b-76f163edd204-kube-api-access-s96f7\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.813607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.811810 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63eff88-dd24-4f72-ac4b-76f163edd204-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.813607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.811901 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.818266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.816410 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63eff88-dd24-4f72-ac4b-76f163edd204-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.827567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.823479 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.832574 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.830763 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.840083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.840051 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s96f7\" (UniqueName: \"kubernetes.io/projected/e63eff88-dd24-4f72-ac4b-76f163edd204-kube-api-access-s96f7\") pod \"maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:07.882631 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:07.882557 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:08.210840 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:08.210805 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n"] Oct 30 02:08:08.212014 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:08:08.211982 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63eff88_dd24_4f72_ac4b_76f163edd204.slice/crio-6e66bea45dd6aec02335b25ad4e11bc0038076a291d9f4e49d8e00985fb1df17 WatchSource:0}: Error finding container 6e66bea45dd6aec02335b25ad4e11bc0038076a291d9f4e49d8e00985fb1df17: Status 404 returned error can't find the container with id 6e66bea45dd6aec02335b25ad4e11bc0038076a291d9f4e49d8e00985fb1df17 Oct 30 02:08:08.527153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:08.527056 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" event={"ID":"e63eff88-dd24-4f72-ac4b-76f163edd204","Type":"ContainerStarted","Data":"6e66bea45dd6aec02335b25ad4e11bc0038076a291d9f4e49d8e00985fb1df17"} Oct 30 02:08:09.435221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:09.435189 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:08:09.537977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:09.537934 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" event={"ID":"e63eff88-dd24-4f72-ac4b-76f163edd204","Type":"ContainerStarted","Data":"7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344"} Oct 30 02:08:09.539046 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:09.539021 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:09.587257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:09.587198 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" podStartSLOduration=2.214788449 podStartE2EDuration="2.587181448s" totalImagesPullingTime="372.392999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:09.586042171 +0000 UTC m=+2368.088974391" watchObservedRunningTime="2026-10-30 02:08:09.587181448 +0000 UTC m=+2368.090113696" Oct 30 02:08:11.348372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:11.348329 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z"] Oct 30 02:08:11.377358 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:11.377320 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg"] Oct 30 02:08:16.348724 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.348674 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" podUID="6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4" containerName="payload-processing" containerID="cri-o://d686cb58960d347cf7cf7c6b4c6c3f09665aa493dea5c9ac4dd6fd2e51f35c8a" gracePeriod=25 Oct 30 02:08:16.377773 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.377736 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" podUID="a0e59e78-5498-4688-b4c7-e3a6911b6511" containerName="payload-pre-processing" containerID="cri-o://34730face63504df59df485ecd98b8b58eeb57a7f41c70dab6a8861a12af884b" gracePeriod=25 Oct 30 02:08:16.559160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.558476 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:08:16.651788 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.651740 5035 generic.go:356] "Generic (PLEG): container finished" podID="6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4" containerID="d686cb58960d347cf7cf7c6b4c6c3f09665aa493dea5c9ac4dd6fd2e51f35c8a" exitCode=0 Oct 30 02:08:16.651978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.651951 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" event={"ID":"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4","Type":"ContainerDied","Data":"d686cb58960d347cf7cf7c6b4c6c3f09665aa493dea5c9ac4dd6fd2e51f35c8a"} Oct 30 02:08:16.654558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.654512 5035 generic.go:356] "Generic (PLEG): container finished" podID="a0e59e78-5498-4688-b4c7-e3a6911b6511" containerID="34730face63504df59df485ecd98b8b58eeb57a7f41c70dab6a8861a12af884b" exitCode=0 Oct 30 02:08:16.654971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.654831 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" event={"ID":"a0e59e78-5498-4688-b4c7-e3a6911b6511","Type":"ContainerDied","Data":"34730face63504df59df485ecd98b8b58eeb57a7f41c70dab6a8861a12af884b"} Oct 30 02:08:16.731834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.731795 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:08:16.822780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.822730 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-kube-api-access-j58g5\" (UniqueName: \"kubernetes.io/projected/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-kube-api-access-j58g5\") pod \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\" (UID: \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\") " Oct 30 02:08:16.822867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.822827 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-plugins-config-volume\") pod \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\" (UID: \"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4\") " Oct 30 02:08:16.834289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.825048 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-plugins-config-volume" pod "6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4" (UID: "6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:16.834289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.827812 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-kube-api-access-j58g5" pod "6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4" (UID: "6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4"). InnerVolumeSpecName "kube-api-access-j58g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:16.848407 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.848371 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:08:16.924580 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.924431 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j58g5\" (UniqueName: \"kubernetes.io/projected/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-kube-api-access-j58g5\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:16.924580 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:16.924486 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:17.025869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.025828 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e59e78-5498-4688-b4c7-e3a6911b6511-kube-api-access-5vqjl\" (UniqueName: \"kubernetes.io/projected/a0e59e78-5498-4688-b4c7-e3a6911b6511-kube-api-access-5vqjl\") pod \"a0e59e78-5498-4688-b4c7-e3a6911b6511\" (UID: \"a0e59e78-5498-4688-b4c7-e3a6911b6511\") " Oct 30 02:08:17.026073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.026063 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e59e78-5498-4688-b4c7-e3a6911b6511-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59e78-5498-4688-b4c7-e3a6911b6511-plugins-config-volume\") pod \"a0e59e78-5498-4688-b4c7-e3a6911b6511\" (UID: \"a0e59e78-5498-4688-b4c7-e3a6911b6511\") " Oct 30 02:08:17.026463 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.026434 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e59e78-5498-4688-b4c7-e3a6911b6511-plugins-config-volume" pod "a0e59e78-5498-4688-b4c7-e3a6911b6511" (UID: "a0e59e78-5498-4688-b4c7-e3a6911b6511"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:17.028140 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.028106 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e59e78-5498-4688-b4c7-e3a6911b6511-kube-api-access-5vqjl" pod "a0e59e78-5498-4688-b4c7-e3a6911b6511" (UID: "a0e59e78-5498-4688-b4c7-e3a6911b6511"). InnerVolumeSpecName "kube-api-access-5vqjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:17.127607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.127568 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vqjl\" (UniqueName: \"kubernetes.io/projected/a0e59e78-5498-4688-b4c7-e3a6911b6511-kube-api-access-5vqjl\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:17.127607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.127601 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59e78-5498-4688-b4c7-e3a6911b6511-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:17.631928 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:08:17.631816 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:08:17.756520 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.756479 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" Oct 30 02:08:17.757275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.757243 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z" event={"ID":"6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4","Type":"ContainerDied","Data":"21943a5249ea20004742c922f58704322223c6893398ee4a935ce8f9703d06a5"} Oct 30 02:08:17.757502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.757475 5035 scope.go:122] "RemoveContainer" containerID="d686cb58960d347cf7cf7c6b4c6c3f09665aa493dea5c9ac4dd6fd2e51f35c8a" Oct 30 02:08:17.786383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.785398 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" event={"ID":"a0e59e78-5498-4688-b4c7-e3a6911b6511","Type":"ContainerDied","Data":"caa4b11b81982b7a1cbdee7c6f63849f668336580a6ce7237c2bfc0699d0f828"} Oct 30 02:08:17.786383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.785514 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg" Oct 30 02:08:17.814758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.807083 5035 scope.go:122] "RemoveContainer" containerID="34730face63504df59df485ecd98b8b58eeb57a7f41c70dab6a8861a12af884b" Oct 30 02:08:17.857380 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.857184 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg"] Oct 30 02:08:17.870794 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.870760 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3ad281-5747c794bc-9tdgg"] Oct 30 02:08:17.901405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.901294 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z"] Oct 30 02:08:17.905030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:17.904988 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3ad281-7cdfb786fb-xls4z"] Oct 30 02:08:19.644798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:19.644754 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4" path="/var/lib/kubelet/pods/6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4/volumes" Oct 30 02:08:19.645453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:19.645428 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e59e78-5498-4688-b4c7-e3a6911b6511" path="/var/lib/kubelet/pods/a0e59e78-5498-4688-b4c7-e3a6911b6511/volumes" Oct 30 02:08:21.438849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.438812 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5"] Oct 30 02:08:21.439394 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.439116 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" podUID="7e8d8d03-2379-40d3-a54b-82a119a35c9a" containerName="maas-api" containerID="cri-o://67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940" gracePeriod=30 Oct 30 02:08:21.795313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.795284 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:08:21.822483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.821679 5035 generic.go:356] "Generic (PLEG): container finished" podID="7e8d8d03-2379-40d3-a54b-82a119a35c9a" containerID="67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940" exitCode=0 Oct 30 02:08:21.822483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.821827 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" Oct 30 02:08:21.822483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.822114 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" event={"ID":"7e8d8d03-2379-40d3-a54b-82a119a35c9a","Type":"ContainerDied","Data":"67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940"} Oct 30 02:08:21.822483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.822182 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5" event={"ID":"7e8d8d03-2379-40d3-a54b-82a119a35c9a","Type":"ContainerDied","Data":"75d5d167eeb9b2110f9dce64fd9c150357d251d00bda162ae91f03bdc4b32860"} Oct 30 02:08:21.822483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.822220 5035 scope.go:122] "RemoveContainer" containerID="67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940" Oct 30 02:08:21.835189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.835162 5035 scope.go:122] "RemoveContainer" containerID="67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940" Oct 30 02:08:21.835573 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:08:21.835550 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940\": container with ID starting with 67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940 not found: ID does not exist" containerID="67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940" Oct 30 02:08:21.835644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.835586 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940"} err="failed to get container status \"67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940\": rpc error: code = NotFound desc = could not find container \"67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940\": container with ID starting with 67babb152bcabbe4f335ee705b00354aed8f025dfcb61a9beecb264380b42940 not found: ID does not exist" Oct 30 02:08:21.839036 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.839010 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8d8d03-2379-40d3-a54b-82a119a35c9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8d03-2379-40d3-a54b-82a119a35c9a-openshift-service-ca\") pod \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " Oct 30 02:08:21.839154 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.839073 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-metrics-certs\") pod \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " Oct 30 02:08:21.839262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.839251 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8d8d03-2379-40d3-a54b-82a119a35c9a-kube-api-access-wmlw6\" (UniqueName: \"kubernetes.io/projected/7e8d8d03-2379-40d3-a54b-82a119a35c9a-kube-api-access-wmlw6\") pod \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " Oct 30 02:08:21.839297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.839285 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-maas-api-tls\") pod \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\" (UID: \"7e8d8d03-2379-40d3-a54b-82a119a35c9a\") " Oct 30 02:08:21.839487 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.839446 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8d8d03-2379-40d3-a54b-82a119a35c9a-openshift-service-ca" pod "7e8d8d03-2379-40d3-a54b-82a119a35c9a" (UID: "7e8d8d03-2379-40d3-a54b-82a119a35c9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:21.839885 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.839859 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8d8d03-2379-40d3-a54b-82a119a35c9a-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:21.841696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.841661 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-metrics-certs" pod "7e8d8d03-2379-40d3-a54b-82a119a35c9a" (UID: "7e8d8d03-2379-40d3-a54b-82a119a35c9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:08:21.841947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.841916 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8d8d03-2379-40d3-a54b-82a119a35c9a-kube-api-access-wmlw6" pod "7e8d8d03-2379-40d3-a54b-82a119a35c9a" (UID: "7e8d8d03-2379-40d3-a54b-82a119a35c9a"). InnerVolumeSpecName "kube-api-access-wmlw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:21.842257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.842229 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-maas-api-tls" pod "7e8d8d03-2379-40d3-a54b-82a119a35c9a" (UID: "7e8d8d03-2379-40d3-a54b-82a119a35c9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:08:21.941217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.941174 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmlw6\" (UniqueName: \"kubernetes.io/projected/7e8d8d03-2379-40d3-a54b-82a119a35c9a-kube-api-access-wmlw6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:21.941217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.941220 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:21.941437 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:21.941229 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8d8d03-2379-40d3-a54b-82a119a35c9a-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:22.174056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:22.174010 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5"] Oct 30 02:08:22.187852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:22.187815 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3ad281-55b8fdbf-lknk5"] Oct 30 02:08:23.118818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.118731 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9"] Oct 30 02:08:23.119437 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.119422 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4" containerName="payload-processing" Oct 30 02:08:23.119499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.119438 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4" containerName="payload-processing" Oct 30 02:08:23.119499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.119461 5035 cpu_manager.go:414] "containerMap: removing container" podUID="7e8d8d03-2379-40d3-a54b-82a119a35c9a" containerName="maas-api" Oct 30 02:08:23.119499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.119466 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8d8d03-2379-40d3-a54b-82a119a35c9a" containerName="maas-api" Oct 30 02:08:23.119499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.119479 5035 cpu_manager.go:414] "containerMap: removing container" podUID="a0e59e78-5498-4688-b4c7-e3a6911b6511" containerName="payload-pre-processing" Oct 30 02:08:23.119499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.119484 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e59e78-5498-4688-b4c7-e3a6911b6511" containerName="payload-pre-processing" Oct 30 02:08:23.119704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.119610 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8d8d03-2379-40d3-a54b-82a119a35c9a" containerName="maas-api" Oct 30 02:08:23.119704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.119621 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e59e78-5498-4688-b4c7-e3a6911b6511" containerName="payload-pre-processing" Oct 30 02:08:23.119704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.119634 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcf7094-7cf6-4adb-ad15-b80d40e4a1e4" containerName="payload-processing" Oct 30 02:08:23.124784 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.124758 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:23.144450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.144418 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9"] Oct 30 02:08:23.153802 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.153773 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-594253\"" Oct 30 02:08:23.162802 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.162757 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xjk\" (UniqueName: \"kubernetes.io/projected/02fd08d5-b5af-4b0c-9a55-e789cd75e036-kube-api-access-b6xjk\") pod \"payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9\" (UID: \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:23.163032 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.162985 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fd08d5-b5af-4b0c-9a55-e789cd75e036-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9\" (UID: \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:23.169522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.169482 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj"] Oct 30 02:08:23.174180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.174150 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:23.192400 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.191958 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-594253-dockercfg-mxd98\"" Oct 30 02:08:23.195386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.195354 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj"] Oct 30 02:08:23.283296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.283257 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4r4\" (UniqueName: \"kubernetes.io/projected/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-kube-api-access-8t4r4\") pod \"payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj\" (UID: \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:23.301619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.296869 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fd08d5-b5af-4b0c-9a55-e789cd75e036-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9\" (UID: \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:23.313571 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.303119 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xjk\" (UniqueName: \"kubernetes.io/projected/02fd08d5-b5af-4b0c-9a55-e789cd75e036-kube-api-access-b6xjk\") pod \"payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9\" (UID: \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:23.315727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.315695 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj\" (UID: \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:23.316171 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.302824 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fd08d5-b5af-4b0c-9a55-e789cd75e036-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9\" (UID: \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:23.347574 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.346273 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xjk\" (UniqueName: \"kubernetes.io/projected/02fd08d5-b5af-4b0c-9a55-e789cd75e036-kube-api-access-b6xjk\") pod \"payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9\" (UID: \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:23.417810 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.417691 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj\" (UID: \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:23.417810 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.417781 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4r4\" (UniqueName: \"kubernetes.io/projected/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-kube-api-access-8t4r4\") pod \"payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj\" (UID: \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:23.418510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.418481 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj\" (UID: \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:23.442738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.442624 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:23.443446 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.443415 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4r4\" (UniqueName: \"kubernetes.io/projected/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-kube-api-access-8t4r4\") pod \"payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj\" (UID: \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:23.495807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.495767 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:23.648904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.648855 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8d8d03-2379-40d3-a54b-82a119a35c9a" path="/var/lib/kubelet/pods/7e8d8d03-2379-40d3-a54b-82a119a35c9a/volumes" Oct 30 02:08:23.758005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.757975 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9"] Oct 30 02:08:23.759111 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:08:23.759079 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fd08d5_b5af_4b0c_9a55_e789cd75e036.slice/crio-d3b88cc98aa45ac0cede693e72ca5a403489ccde57d8c13cb74f6e2a10bfb53c WatchSource:0}: Error finding container d3b88cc98aa45ac0cede693e72ca5a403489ccde57d8c13cb74f6e2a10bfb53c: Status 404 returned error can't find the container with id d3b88cc98aa45ac0cede693e72ca5a403489ccde57d8c13cb74f6e2a10bfb53c Oct 30 02:08:23.825509 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:08:23.825471 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350c87ad_eb3c_4b12_bdf4_15bf9ef6a7ad.slice/crio-898d7e39f725f88afaf125e2dfc5bc6012169486520923e0f1aa9b6a6677a1b8 WatchSource:0}: Error finding container 898d7e39f725f88afaf125e2dfc5bc6012169486520923e0f1aa9b6a6677a1b8: Status 404 returned error can't find the container with id 898d7e39f725f88afaf125e2dfc5bc6012169486520923e0f1aa9b6a6677a1b8 Oct 30 02:08:23.825678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.825543 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj"] Oct 30 02:08:23.840126 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.840092 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" event={"ID":"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad","Type":"ContainerStarted","Data":"898d7e39f725f88afaf125e2dfc5bc6012169486520923e0f1aa9b6a6677a1b8"} Oct 30 02:08:23.842012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:23.841979 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" event={"ID":"02fd08d5-b5af-4b0c-9a55-e789cd75e036","Type":"ContainerStarted","Data":"d3b88cc98aa45ac0cede693e72ca5a403489ccde57d8c13cb74f6e2a10bfb53c"} Oct 30 02:08:24.851152 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:24.851098 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" event={"ID":"02fd08d5-b5af-4b0c-9a55-e789cd75e036","Type":"ContainerStarted","Data":"d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9"} Oct 30 02:08:24.851785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:24.851193 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:24.853265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:24.853228 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" event={"ID":"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad","Type":"ContainerStarted","Data":"3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541"} Oct 30 02:08:24.853437 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:24.853329 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:24.896437 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:24.896374 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" podStartSLOduration=1.896360606 podStartE2EDuration="1.896360606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:24.895365253 +0000 UTC m=+2383.398297492" watchObservedRunningTime="2026-10-30 02:08:24.896360606 +0000 UTC m=+2383.399292865" Oct 30 02:08:24.941205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:24.941149 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" podStartSLOduration=1.9411343300000001 podStartE2EDuration="1.94113433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:24.939141624 +0000 UTC m=+2383.442073845" watchObservedRunningTime="2026-10-30 02:08:24.94113433 +0000 UTC m=+2383.444066549" Oct 30 02:08:27.397902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.394088 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595"] Oct 30 02:08:27.397902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.394340 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" podUID="0690fb5d-2223-4310-a044-3f179c073ede" containerName="maas-api" containerID="cri-o://f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5" gracePeriod=30 Oct 30 02:08:27.698145 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.698113 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:08:27.810180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.809995 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-maas-api-tls\") pod \"0690fb5d-2223-4310-a044-3f179c073ede\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " Oct 30 02:08:27.810180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.810136 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0690fb5d-2223-4310-a044-3f179c073ede-kube-api-access-g9msn\" (UniqueName: \"kubernetes.io/projected/0690fb5d-2223-4310-a044-3f179c073ede-kube-api-access-g9msn\") pod \"0690fb5d-2223-4310-a044-3f179c073ede\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " Oct 30 02:08:27.810424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.810360 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-metrics-certs\") pod \"0690fb5d-2223-4310-a044-3f179c073ede\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " Oct 30 02:08:27.810519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.810479 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0690fb5d-2223-4310-a044-3f179c073ede-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0690fb5d-2223-4310-a044-3f179c073ede-openshift-service-ca\") pod \"0690fb5d-2223-4310-a044-3f179c073ede\" (UID: \"0690fb5d-2223-4310-a044-3f179c073ede\") " Oct 30 02:08:27.838556 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.838307 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0690fb5d-2223-4310-a044-3f179c073ede-openshift-service-ca" pod "0690fb5d-2223-4310-a044-3f179c073ede" (UID: "0690fb5d-2223-4310-a044-3f179c073ede"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:27.850612 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.847255 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0690fb5d-2223-4310-a044-3f179c073ede-kube-api-access-g9msn" pod "0690fb5d-2223-4310-a044-3f179c073ede" (UID: "0690fb5d-2223-4310-a044-3f179c073ede"). InnerVolumeSpecName "kube-api-access-g9msn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:27.861778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.861729 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-metrics-certs" pod "0690fb5d-2223-4310-a044-3f179c073ede" (UID: "0690fb5d-2223-4310-a044-3f179c073ede"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:08:27.883453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.883363 5035 generic.go:356] "Generic (PLEG): container finished" podID="0690fb5d-2223-4310-a044-3f179c073ede" containerID="f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5" exitCode=0 Oct 30 02:08:27.883740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.883715 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" event={"ID":"0690fb5d-2223-4310-a044-3f179c073ede","Type":"ContainerDied","Data":"f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5"} Oct 30 02:08:27.883869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.883777 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" event={"ID":"0690fb5d-2223-4310-a044-3f179c073ede","Type":"ContainerDied","Data":"3c4b105ddbb47bb2cd6c9529a11399a75efd8bea95db2b122ac13c029b3f4bae"} Oct 30 02:08:27.883869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.883809 5035 scope.go:122] "RemoveContainer" containerID="f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5" Oct 30 02:08:27.884101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.884004 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595" Oct 30 02:08:27.903078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.902999 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-maas-api-tls" pod "0690fb5d-2223-4310-a044-3f179c073ede" (UID: "0690fb5d-2223-4310-a044-3f179c073ede"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:08:27.908102 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.908072 5035 scope.go:122] "RemoveContainer" containerID="f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5" Oct 30 02:08:27.908466 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:08:27.908436 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5\": container with ID starting with f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5 not found: ID does not exist" containerID="f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5" Oct 30 02:08:27.908523 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.908472 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5"} err="failed to get container status \"f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5\": rpc error: code = NotFound desc = could not find container \"f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5\": container with ID starting with f7f967a9ec5b7c3f0e214a3535e66a5e484dfe41dd6ca658c31b753d218f62c5 not found: ID does not exist" Oct 30 02:08:27.913304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.913195 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:27.913304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.913236 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0690fb5d-2223-4310-a044-3f179c073ede-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:27.913304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.913252 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0690fb5d-2223-4310-a044-3f179c073ede-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:27.913304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:27.913269 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9msn\" (UniqueName: \"kubernetes.io/projected/0690fb5d-2223-4310-a044-3f179c073ede-kube-api-access-g9msn\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:28.234944 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:28.234906 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595"] Oct 30 02:08:28.238436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:28.238402 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b4a561-78675997b9-s2595"] Oct 30 02:08:29.643981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:29.643943 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0690fb5d-2223-4310-a044-3f179c073ede" path="/var/lib/kubelet/pods/0690fb5d-2223-4310-a044-3f179c073ede/volumes" Oct 30 02:08:35.863888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:35.863855 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:08:35.864412 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:35.863947 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:08:36.881519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:36.881466 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv"] Oct 30 02:08:36.882082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:36.881839 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" containerID="cri-o://3c3f68a72353b32ca0c3f598be510ef3f8790f79c73418d439b4d053cac44ad3" gracePeriod=30 Oct 30 02:08:38.813495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:38.813112 5035 patch_prober.go:29] interesting pod/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:08:38.813495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:38.813179 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:08:40.878336 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.878295 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms"] Oct 30 02:08:40.879046 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.879030 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0690fb5d-2223-4310-a044-3f179c073ede" containerName="maas-api" Oct 30 02:08:40.879103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.879047 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0690fb5d-2223-4310-a044-3f179c073ede" containerName="maas-api" Oct 30 02:08:40.879162 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.879152 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0690fb5d-2223-4310-a044-3f179c073ede" containerName="maas-api" Oct 30 02:08:40.882814 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.882793 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:40.909298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.909259 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms"] Oct 30 02:08:40.917185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.917148 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-594253-serving-cert\"" Oct 30 02:08:40.960999 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.960956 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:40.961449 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.961428 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2qf7\" (UniqueName: \"kubernetes.io/projected/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-kube-api-access-h2qf7\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:40.961698 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.961680 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:40.961880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:40.961851 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.063479 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.063434 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2qf7\" (UniqueName: \"kubernetes.io/projected/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-kube-api-access-h2qf7\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.063713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.063572 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.063713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.063599 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.063713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.063636 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.063885 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:08:41.063724 5035 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-serving-cert: secret "maas-api-e2e-shared-b-w3-594253-serving-cert" not found Oct 30 02:08:41.063885 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:08:41.063788 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls podName:92c1cf7d-f6ca-43b7-aff3-99b29fd127c2 nodeName:}" failed. No retries permitted until 2026-10-30 02:08:41.56376874 +0000 UTC m=+2400.066700945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls") pod "maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" (UID: "92c1cf7d-f6ca-43b7-aff3-99b29fd127c2") : secret "maas-api-e2e-shared-b-w3-594253-serving-cert" not found Oct 30 02:08:41.064426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.064396 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.066254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.066230 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.093373 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.093336 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2qf7\" (UniqueName: \"kubernetes.io/projected/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-kube-api-access-h2qf7\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.571835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.571798 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.574481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.574457 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:41.798293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:41.798237 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:42.117816 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.117102 5035 generic.go:356] "Generic (PLEG): container finished" podID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerID="3c3f68a72353b32ca0c3f598be510ef3f8790f79c73418d439b4d053cac44ad3" exitCode=0 Oct 30 02:08:42.117816 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.117385 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" event={"ID":"ef3cea51-d79d-4f51-9ebb-bd28f0335889","Type":"ContainerDied","Data":"3c3f68a72353b32ca0c3f598be510ef3f8790f79c73418d439b4d053cac44ad3"} Oct 30 02:08:42.117816 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.117580 5035 scope.go:122] "RemoveContainer" containerID="a395e1a73e2180088d7146685523ee09520eb874b6ae2526a22050dd337cce6b" Oct 30 02:08:42.155455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.155326 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms"] Oct 30 02:08:42.156610 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:08:42.156571 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c1cf7d_f6ca_43b7_aff3_99b29fd127c2.slice/crio-dedb443ad837048bfdcae042ecc640936c3134a6deb1565570b21aba3f8a835f WatchSource:0}: Error finding container dedb443ad837048bfdcae042ecc640936c3134a6deb1565570b21aba3f8a835f: Status 404 returned error can't find the container with id dedb443ad837048bfdcae042ecc640936c3134a6deb1565570b21aba3f8a835f Oct 30 02:08:42.313436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.313407 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:08:42.396314 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396226 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istiod-ca-cert\") pod \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " Oct 30 02:08:42.396314 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396296 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-podinfo\") pod \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " Oct 30 02:08:42.396599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396372 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-kube-api-access-kw7j7\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-kube-api-access-kw7j7\") pod \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " Oct 30 02:08:42.396599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396424 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-credential-socket\") pod \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " Oct 30 02:08:42.396599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396493 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-data\") pod \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " Oct 30 02:08:42.396782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396619 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-certs\") pod \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " Oct 30 02:08:42.396782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396735 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-token\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-token\") pod \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " Oct 30 02:08:42.396903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396786 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-envoy\") pod \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " Oct 30 02:08:42.396903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396820 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istiod-ca-cert" pod "ef3cea51-d79d-4f51-9ebb-bd28f0335889" (UID: "ef3cea51-d79d-4f51-9ebb-bd28f0335889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:42.396903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396828 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-socket\") pod \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\" (UID: \"ef3cea51-d79d-4f51-9ebb-bd28f0335889\") " Oct 30 02:08:42.397051 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.396942 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-data" pod "ef3cea51-d79d-4f51-9ebb-bd28f0335889" (UID: "ef3cea51-d79d-4f51-9ebb-bd28f0335889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:42.397051 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.397012 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-socket" pod "ef3cea51-d79d-4f51-9ebb-bd28f0335889" (UID: "ef3cea51-d79d-4f51-9ebb-bd28f0335889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:42.397263 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.397235 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-certs" pod "ef3cea51-d79d-4f51-9ebb-bd28f0335889" (UID: "ef3cea51-d79d-4f51-9ebb-bd28f0335889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:42.397585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.397560 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:42.397690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.397586 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:42.397690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.397599 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:42.397690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.397612 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:42.397690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.397680 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-credential-socket" pod "ef3cea51-d79d-4f51-9ebb-bd28f0335889" (UID: "ef3cea51-d79d-4f51-9ebb-bd28f0335889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:42.399108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.399074 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-token" pod "ef3cea51-d79d-4f51-9ebb-bd28f0335889" (UID: "ef3cea51-d79d-4f51-9ebb-bd28f0335889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:42.399108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.399093 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-kube-api-access-kw7j7" pod "ef3cea51-d79d-4f51-9ebb-bd28f0335889" (UID: "ef3cea51-d79d-4f51-9ebb-bd28f0335889"). InnerVolumeSpecName "kube-api-access-kw7j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:42.399286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.399235 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-podinfo" pod "ef3cea51-d79d-4f51-9ebb-bd28f0335889" (UID: "ef3cea51-d79d-4f51-9ebb-bd28f0335889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:08:42.399376 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.399359 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-envoy" pod "ef3cea51-d79d-4f51-9ebb-bd28f0335889" (UID: "ef3cea51-d79d-4f51-9ebb-bd28f0335889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:42.499224 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.499187 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw7j7\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-kube-api-access-kw7j7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:42.499224 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.499222 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:42.499426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.499234 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:42.499426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.499246 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:42.499426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:42.499254 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3cea51-d79d-4f51-9ebb-bd28f0335889-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:08:43.126816 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.126766 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" event={"ID":"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2","Type":"ContainerStarted","Data":"891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93"} Oct 30 02:08:43.127322 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.126838 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" event={"ID":"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2","Type":"ContainerStarted","Data":"dedb443ad837048bfdcae042ecc640936c3134a6deb1565570b21aba3f8a835f"} Oct 30 02:08:43.127322 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.126869 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:43.128585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.128518 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" event={"ID":"ef3cea51-d79d-4f51-9ebb-bd28f0335889","Type":"ContainerDied","Data":"22972156251e181c71960f8619998dc83e732acdb48c17a5cdeef35eb3ca2fda"} Oct 30 02:08:43.128585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.128580 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv" Oct 30 02:08:43.128745 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.128615 5035 scope.go:122] "RemoveContainer" containerID="3c3f68a72353b32ca0c3f598be510ef3f8790f79c73418d439b4d053cac44ad3" Oct 30 02:08:43.184194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.184138 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" podStartSLOduration=2.87721854 podStartE2EDuration="3.184116447s" totalImagesPullingTime="306.897907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:43.181288473 +0000 UTC m=+2401.684220692" watchObservedRunningTime="2026-10-30 02:08:43.184116447 +0000 UTC m=+2401.687048668" Oct 30 02:08:43.238413 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.238374 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv"] Oct 30 02:08:43.251954 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.251905 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b4a561-openshift-default-65f79b4c6f-mnltv"] Oct 30 02:08:43.644600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.644516 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" path="/var/lib/kubelet/pods/ef3cea51-d79d-4f51-9ebb-bd28f0335889/volumes" Oct 30 02:08:43.718617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.718587 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:08:43.735029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.734998 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:08:43.755751 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.755725 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:08:43.771303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:43.771204 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:08:47.631399 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:08:47.631358 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:08:48.859731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.859690 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz"] Oct 30 02:08:48.860673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.860652 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" Oct 30 02:08:48.860754 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.860674 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" Oct 30 02:08:48.860891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.860879 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" Oct 30 02:08:48.860929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.860897 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" Oct 30 02:08:48.879927 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.879874 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:08:48.901256 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.901224 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz"] Oct 30 02:08:48.904246 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.904217 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-584ec5\"" Oct 30 02:08:48.918216 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.918178 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-584ec5-dockercfg-7twsn\"" Oct 30 02:08:48.924191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.924153 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5"] Oct 30 02:08:48.925202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.925177 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" Oct 30 02:08:48.925202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.925201 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3cea51-d79d-4f51-9ebb-bd28f0335889" containerName="istio-proxy" Oct 30 02:08:48.929384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.929350 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:08:48.951591 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:48.948809 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5"] Oct 30 02:08:49.157582 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.157455 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:08:49.160937 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.160899 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b57914-820a-470a-bb14-34c53a03840a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz\" (UID: \"f4b57914-820a-470a-bb14-34c53a03840a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:08:49.161129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.160950 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hktb\" (UniqueName: \"kubernetes.io/projected/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-kube-api-access-8hktb\") pod \"payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5\" (UID: \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:08:49.161129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.161036 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5\" (UID: \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:08:49.161129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.161079 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd87r\" (UniqueName: \"kubernetes.io/projected/f4b57914-820a-470a-bb14-34c53a03840a-kube-api-access-jd87r\") pod \"payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz\" (UID: \"f4b57914-820a-470a-bb14-34c53a03840a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:08:49.262642 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.262596 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b57914-820a-470a-bb14-34c53a03840a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz\" (UID: \"f4b57914-820a-470a-bb14-34c53a03840a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:08:49.262863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.262666 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hktb\" (UniqueName: \"kubernetes.io/projected/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-kube-api-access-8hktb\") pod \"payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5\" (UID: \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:08:49.262863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.262792 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5\" (UID: \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:08:49.262863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.262841 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd87r\" (UniqueName: \"kubernetes.io/projected/f4b57914-820a-470a-bb14-34c53a03840a-kube-api-access-jd87r\") pod \"payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz\" (UID: \"f4b57914-820a-470a-bb14-34c53a03840a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:08:49.264373 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.264335 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5\" (UID: \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:08:49.264862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.264835 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b57914-820a-470a-bb14-34c53a03840a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz\" (UID: \"f4b57914-820a-470a-bb14-34c53a03840a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:08:49.295595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.294585 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hktb\" (UniqueName: \"kubernetes.io/projected/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-kube-api-access-8hktb\") pod \"payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5\" (UID: \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:08:49.295595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.295072 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd87r\" (UniqueName: \"kubernetes.io/projected/f4b57914-820a-470a-bb14-34c53a03840a-kube-api-access-jd87r\") pod \"payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz\" (UID: \"f4b57914-820a-470a-bb14-34c53a03840a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:08:49.515205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.515154 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:08:49.564883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.564840 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:08:49.855941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:49.855895 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz"] Oct 30 02:08:49.856608 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:08:49.856570 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b57914_820a_470a_bb14_34c53a03840a.slice/crio-3668c389b4b185a9aa3e7ab17b745ed3452e4b8889b358e8d4a582129a109605 WatchSource:0}: Error finding container 3668c389b4b185a9aa3e7ab17b745ed3452e4b8889b358e8d4a582129a109605: Status 404 returned error can't find the container with id 3668c389b4b185a9aa3e7ab17b745ed3452e4b8889b358e8d4a582129a109605 Oct 30 02:08:50.012612 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:08:50.012241 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab68786d_fb38_4ba1_bbaf_a5bbfcd96765.slice/crio-6469b597719b68ca2453a66436927737d0a3a796e54ddc3634578ff1b0b7d603 WatchSource:0}: Error finding container 6469b597719b68ca2453a66436927737d0a3a796e54ddc3634578ff1b0b7d603: Status 404 returned error can't find the container with id 6469b597719b68ca2453a66436927737d0a3a796e54ddc3634578ff1b0b7d603 Oct 30 02:08:50.014803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:50.014462 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5"] Oct 30 02:08:50.214578 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:50.214522 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" event={"ID":"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765","Type":"ContainerStarted","Data":"bec2352d71cf6d8b40f480c2369fa6cc62342d49053711cf6e4147cf13ed0209"} Oct 30 02:08:50.214765 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:50.214607 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:08:50.214765 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:50.214626 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" event={"ID":"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765","Type":"ContainerStarted","Data":"6469b597719b68ca2453a66436927737d0a3a796e54ddc3634578ff1b0b7d603"} Oct 30 02:08:50.216969 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:50.216934 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" event={"ID":"f4b57914-820a-470a-bb14-34c53a03840a","Type":"ContainerStarted","Data":"6f741b6659090c153a51a1480df61c017284033e30497216babb87f6b53e6c43"} Oct 30 02:08:50.217145 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:50.216995 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" event={"ID":"f4b57914-820a-470a-bb14-34c53a03840a","Type":"ContainerStarted","Data":"3668c389b4b185a9aa3e7ab17b745ed3452e4b8889b358e8d4a582129a109605"} Oct 30 02:08:50.217145 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:50.217048 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:08:50.301393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:50.301331 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" podStartSLOduration=2.301315938 podStartE2EDuration="2.301315938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:50.297990585 +0000 UTC m=+2408.800922810" watchObservedRunningTime="2026-10-30 02:08:50.301315938 +0000 UTC m=+2408.804248185" Oct 30 02:08:50.361357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:50.361218 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" podStartSLOduration=2.361201237 podStartE2EDuration="2.361201237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:50.357363958 +0000 UTC m=+2408.860296177" watchObservedRunningTime="2026-10-30 02:08:50.361201237 +0000 UTC m=+2408.864133470" Oct 30 02:08:57.276183 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:57.276138 5035 generic.go:356] "Generic (PLEG): container finished" podID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerID="5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5" exitCode=0 Oct 30 02:08:57.276183 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:57.276172 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" event={"ID":"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3","Type":"ContainerDied","Data":"5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5"} Oct 30 02:08:57.276761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:57.276661 5035 scope.go:122] "RemoveContainer" containerID="5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5" Oct 30 02:08:57.278438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:57.278406 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:08:57.278586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:57.278489 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:08:57.278586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:57.278523 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:08:58.301842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:58.301800 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" event={"ID":"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3","Type":"ContainerStarted","Data":"b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019"} Oct 30 02:08:58.430591 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:58.430553 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:08:58.431930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:58.431891 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:08:58.432102 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:58.431987 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 02:08:59.430423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:59.430376 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:08:59.431101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:08:59.430451 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 02:09:00.288553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:00.275644 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:09:00.430029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:00.429992 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:09:00.430258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:00.430047 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 02:09:01.227775 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:01.227736 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:09:01.228255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:01.227997 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:09:01.430654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:01.430616 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:09:01.430852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:01.430683 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 02:09:02.195170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.195126 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc"] Oct 30 02:09:02.201155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.201122 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc"] Oct 30 02:09:02.201349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.201229 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:02.218539 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.218495 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a1028d-dockercfg-9b49r\"" Oct 30 02:09:02.220212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.220188 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a1028d\"" Oct 30 02:09:02.315057 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.314424 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rk5\" (UniqueName: \"kubernetes.io/projected/6cc33552-ba67-42dd-bedf-73513dda1dac-kube-api-access-98rk5\") pod \"payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc\" (UID: \"6cc33552-ba67-42dd-bedf-73513dda1dac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:02.315057 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.314918 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc33552-ba67-42dd-bedf-73513dda1dac-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc\" (UID: \"6cc33552-ba67-42dd-bedf-73513dda1dac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:02.436277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.436210 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc33552-ba67-42dd-bedf-73513dda1dac-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc\" (UID: \"6cc33552-ba67-42dd-bedf-73513dda1dac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:02.436507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.436316 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rk5\" (UniqueName: \"kubernetes.io/projected/6cc33552-ba67-42dd-bedf-73513dda1dac-kube-api-access-98rk5\") pod \"payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc\" (UID: \"6cc33552-ba67-42dd-bedf-73513dda1dac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:02.436507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.436385 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:09:02.436507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.436456 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 02:09:02.442592 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.437448 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc33552-ba67-42dd-bedf-73513dda1dac-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc\" (UID: \"6cc33552-ba67-42dd-bedf-73513dda1dac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:02.461977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.461943 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rk5\" (UniqueName: \"kubernetes.io/projected/6cc33552-ba67-42dd-bedf-73513dda1dac-kube-api-access-98rk5\") pod \"payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc\" (UID: \"6cc33552-ba67-42dd-bedf-73513dda1dac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:02.518935 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.518891 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:02.906606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:02.905670 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc"] Oct 30 02:09:02.908289 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:09:02.908248 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc33552_ba67_42dd_bedf_73513dda1dac.slice/crio-ddbb50bba020e613b03922095bf3c28718eb20c9dae112a273c5cb9d642f43a3 WatchSource:0}: Error finding container ddbb50bba020e613b03922095bf3c28718eb20c9dae112a273c5cb9d642f43a3: Status 404 returned error can't find the container with id ddbb50bba020e613b03922095bf3c28718eb20c9dae112a273c5cb9d642f43a3 Oct 30 02:09:03.430097 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:03.430057 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 02:09:03.430688 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:03.430120 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 02:09:03.442077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:03.442029 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" event={"ID":"6cc33552-ba67-42dd-bedf-73513dda1dac","Type":"ContainerStarted","Data":"7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433"} Oct 30 02:09:03.442297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:03.442100 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" event={"ID":"6cc33552-ba67-42dd-bedf-73513dda1dac","Type":"ContainerStarted","Data":"ddbb50bba020e613b03922095bf3c28718eb20c9dae112a273c5cb9d642f43a3"} Oct 30 02:09:03.442297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:03.442129 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:03.492579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:03.492496 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" podStartSLOduration=1.492479093 podStartE2EDuration="1.492479093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:03.490433523 +0000 UTC m=+2421.993365744" watchObservedRunningTime="2026-10-30 02:09:03.492479093 +0000 UTC m=+2421.995411358" Oct 30 02:09:04.435683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:04.435644 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:09:04.437064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:04.437038 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:09:09.585671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.585633 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9"] Oct 30 02:09:09.590683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.590656 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.617108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.617064 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9"] Oct 30 02:09:09.627832 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.627793 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-584ec5-serving-cert\"" Oct 30 02:09:09.759372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.759335 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvm4p\" (UniqueName: \"kubernetes.io/projected/053375c1-4f44-4ae1-9272-cbd76b067d5a-kube-api-access-jvm4p\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.759556 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.759389 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.759556 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.759514 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.759707 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.759667 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053375c1-4f44-4ae1-9272-cbd76b067d5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.861445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.861342 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvm4p\" (UniqueName: \"kubernetes.io/projected/053375c1-4f44-4ae1-9272-cbd76b067d5a-kube-api-access-jvm4p\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.861445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.861407 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.861769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.861511 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.861769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.861610 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053375c1-4f44-4ae1-9272-cbd76b067d5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.862784 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.862748 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053375c1-4f44-4ae1-9272-cbd76b067d5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.864573 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.864543 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.864773 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.864670 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.891978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.891937 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvm4p\" (UniqueName: \"kubernetes.io/projected/053375c1-4f44-4ae1-9272-cbd76b067d5a-kube-api-access-jvm4p\") pod \"maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:09.917222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:09.917183 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:10.240458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:10.240420 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9"] Oct 30 02:09:10.241160 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:09:10.241130 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053375c1_4f44_4ae1_9272_cbd76b067d5a.slice/crio-924c64b0b864dbc48c8017e65d0e31e6fcd413e89e715596a6499363d930da75 WatchSource:0}: Error finding container 924c64b0b864dbc48c8017e65d0e31e6fcd413e89e715596a6499363d930da75: Status 404 returned error can't find the container with id 924c64b0b864dbc48c8017e65d0e31e6fcd413e89e715596a6499363d930da75 Oct 30 02:09:10.506306 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:10.506268 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" event={"ID":"053375c1-4f44-4ae1-9272-cbd76b067d5a","Type":"ContainerStarted","Data":"924c64b0b864dbc48c8017e65d0e31e6fcd413e89e715596a6499363d930da75"} Oct 30 02:09:11.514946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:11.514901 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" event={"ID":"053375c1-4f44-4ae1-9272-cbd76b067d5a","Type":"ContainerStarted","Data":"5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416"} Oct 30 02:09:11.515360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:11.515016 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:11.565813 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:11.565758 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" podStartSLOduration=2.24784372 podStartE2EDuration="2.56574164s" totalImagesPullingTime="317.89792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:11.562600427 +0000 UTC m=+2430.065532650" watchObservedRunningTime="2026-10-30 02:09:11.56574164 +0000 UTC m=+2430.068673910" Oct 30 02:09:13.108025 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.107981 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d"] Oct 30 02:09:13.112683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.112660 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.134588 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.134558 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-719029-openshift-default-dockercfg-9jg4v\"" Oct 30 02:09:13.164989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.164953 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d"] Oct 30 02:09:13.195048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.194992 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvkk\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-kube-api-access-rdvkk\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.195274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.195134 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.195274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.195188 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.195424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.195284 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.195490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.195433 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa390600-ff7d-4f05-bc0c-9ada10153b57-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.195490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.195463 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.195705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.195680 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.195807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.195734 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.195807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.195757 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.297584 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.297494 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.297805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.297626 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.297805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.297657 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.297805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.297778 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvkk\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-kube-api-access-rdvkk\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.297984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.297846 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.297984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.297873 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.297984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.297910 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.298160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.298054 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa390600-ff7d-4f05-bc0c-9ada10153b57-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.298160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.298070 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.298160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.298085 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.298363 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.298337 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.298759 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.298732 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.298885 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.298799 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa390600-ff7d-4f05-bc0c-9ada10153b57-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.298948 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.298901 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.300200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.300177 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.300417 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.300397 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.325823 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.325776 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.329358 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.329325 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvkk\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-kube-api-access-rdvkk\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.428769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.428656 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:13.773012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.772977 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d"] Oct 30 02:09:13.778060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.778027 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:09:13.778153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.778100 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:09:13.778153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:13.778131 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:09:14.453648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:14.453612 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:09:14.546890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:14.546853 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" event={"ID":"aa390600-ff7d-4f05-bc0c-9ada10153b57","Type":"ContainerStarted","Data":"6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94"} Oct 30 02:09:14.547103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:14.546912 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" event={"ID":"aa390600-ff7d-4f05-bc0c-9ada10153b57","Type":"ContainerStarted","Data":"b8e3dda40dc7e21f4bc14af8b687ca5fe7d80505f1c7f4b5faee2c704cf2b12b"} Oct 30 02:09:14.603173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:14.603083 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" podStartSLOduration=1.603060597 podStartE2EDuration="1.603060597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:14.595990784 +0000 UTC m=+2433.098923007" watchObservedRunningTime="2026-10-30 02:09:14.603060597 +0000 UTC m=+2433.105992819" Oct 30 02:09:15.428896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:15.428850 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:15.431141 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:15.431109 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 02:09:15.431321 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:15.431161 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 02:09:16.429511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:16.429463 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 02:09:16.430006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:16.429584 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 02:09:17.429488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:17.429440 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 02:09:17.429734 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:17.429508 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 02:09:17.528089 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:17.528057 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:09:17.644697 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:09:17.644650 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:09:18.429725 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:18.429682 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 02:09:18.430147 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:18.429751 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 02:09:19.429509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:19.429473 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 02:09:19.429735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:19.429565 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 02:09:20.429506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:20.429462 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 02:09:20.430084 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:20.429546 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 02:09:21.565271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:21.565228 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:21.565997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:21.565969 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:21.566805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:21.566783 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:09:38.737981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.737929 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd"] Oct 30 02:09:38.743654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.743621 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.761961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.761890 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-01a9cd-openshift-default-dockercfg-gzcbm\"" Oct 30 02:09:38.766736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.766694 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd"] Oct 30 02:09:38.860553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.859331 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-data\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.860553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.859427 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-certs\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.860553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.859570 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7g4\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-kube-api-access-5s7g4\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.860553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.859648 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-credential-socket\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.860553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.859716 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-envoy\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.860553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.859813 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-token\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.860553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.859842 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-socket\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.860553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.859948 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-podinfo\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.860553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.859992 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istiod-ca-cert\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.974656 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-data\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.974109 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-data\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.974801 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-certs\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.974902 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7g4\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-kube-api-access-5s7g4\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.974957 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-credential-socket\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.975069 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-envoy\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.975212 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-token\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.975242 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-socket\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.975355 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-podinfo\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.975550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.975383 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istiod-ca-cert\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:38.976270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.975862 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-socket\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:39.012598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.993367 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istiod-ca-cert\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:39.012598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:38.993711 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-certs\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:39.012598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.002865 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-podinfo\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:39.012598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.004871 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-credential-socket\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:39.012598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.007473 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-envoy\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:39.036547 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.035828 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7g4\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-kube-api-access-5s7g4\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:39.038184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.038120 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-token\") pod \"e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:39.068677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.068636 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:39.436081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.436035 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd"] Oct 30 02:09:39.437436 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:09:39.437401 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f7a1fb_dc50_4f51_ab08_e745a1a1703b.slice/crio-109373491cfe6c612bf9426949af441929bc8d033b7eb63fb82d247983fc6c69 WatchSource:0}: Error finding container 109373491cfe6c612bf9426949af441929bc8d033b7eb63fb82d247983fc6c69: Status 404 returned error can't find the container with id 109373491cfe6c612bf9426949af441929bc8d033b7eb63fb82d247983fc6c69 Oct 30 02:09:39.440570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.440504 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:09:39.440682 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.440617 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:09:39.440682 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.440660 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:09:39.766374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.766336 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" event={"ID":"14f7a1fb-dc50-4f51-ab08-e745a1a1703b","Type":"ContainerStarted","Data":"ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b"} Oct 30 02:09:39.766914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.766396 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" event={"ID":"14f7a1fb-dc50-4f51-ab08-e745a1a1703b","Type":"ContainerStarted","Data":"109373491cfe6c612bf9426949af441929bc8d033b7eb63fb82d247983fc6c69"} Oct 30 02:09:39.823956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:39.823894 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" podStartSLOduration=1.823877796 podStartE2EDuration="1.823877796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:39.82276039 +0000 UTC m=+2458.325692624" watchObservedRunningTime="2026-10-30 02:09:39.823877796 +0000 UTC m=+2458.326810015" Oct 30 02:09:40.069646 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:40.069513 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:40.076170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:40.076141 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:40.773959 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:40.773921 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:40.775022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:09:40.774997 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:09:47.630329 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:09:47.630287 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:10:17.630974 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:10:17.630888 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:10:36.280801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:36.280762 5035 generic.go:356] "Generic (PLEG): container finished" podID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerID="ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b" exitCode=0 Oct 30 02:10:36.281270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:36.280835 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" event={"ID":"14f7a1fb-dc50-4f51-ab08-e745a1a1703b","Type":"ContainerDied","Data":"ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b"} Oct 30 02:10:36.281326 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:36.281308 5035 scope.go:122] "RemoveContainer" containerID="ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b" Oct 30 02:10:36.283110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:36.283081 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:36.283192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:36.283181 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:36.283230 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:36.283212 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:37.300507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:37.300453 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" event={"ID":"14f7a1fb-dc50-4f51-ab08-e745a1a1703b","Type":"ContainerStarted","Data":"ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099"} Oct 30 02:10:38.084015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:38.083960 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:10:38.091769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:38.091718 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:10:38.092005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:38.091796 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 02:10:39.086082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:39.086034 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:10:39.087860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:39.087372 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 02:10:40.071555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:40.070974 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:10:40.071555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:40.071067 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 02:10:40.786555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:40.780642 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:10:41.088619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:41.085117 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:10:41.088619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:41.085182 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 02:10:42.076557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:42.070956 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 02:10:42.076557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:42.071026 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 02:10:43.074739 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:43.074696 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:10:43.075861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:43.075836 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:10:47.632141 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:10:47.632099 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:10:48.038562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.038491 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz"] Oct 30 02:10:48.047641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.047606 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.069178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.069135 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz"] Oct 30 02:10:48.076652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.076617 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-594253-openshift-default-dockercfg-r8hqf\"" Oct 30 02:10:48.164877 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.164825 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-credential-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.165110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.165004 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istiod-ca-cert\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.165177 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.165108 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.165223 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.165198 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-podinfo\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.165269 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.165246 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-data\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.165319 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.165279 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-envoy\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.165368 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.165336 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-certs\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.165425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.165403 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttkpb\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-kube-api-access-ttkpb\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.165475 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.165457 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-token\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.266711 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.266669 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-credential-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.266918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.266772 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istiod-ca-cert\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.266918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.266835 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.266918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.266893 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-podinfo\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.266936 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-data\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.266961 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-envoy\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.266983 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-certs\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.267044 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttkpb\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-kube-api-access-ttkpb\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.267085 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-token\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.267106 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-credential-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.267219 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-socket\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267515 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.267413 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-data\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267515 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.267497 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-certs\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.267715 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.267692 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istiod-ca-cert\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.269462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.269435 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-envoy\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.269674 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.269654 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-podinfo\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.297156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.297073 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttkpb\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-kube-api-access-ttkpb\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.297324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.297306 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-token\") pod \"e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.368439 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.368394 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:48.707482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.707435 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz"] Oct 30 02:10:48.711242 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:10:48.711209 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a935d8_06c6_43cb_bbc6_adf7b9f60dc3.slice/crio-32c7b5362736ef7ade892d01f85cc4d713e9f32a79fcb324452fb9076ccafc5c WatchSource:0}: Error finding container 32c7b5362736ef7ade892d01f85cc4d713e9f32a79fcb324452fb9076ccafc5c: Status 404 returned error can't find the container with id 32c7b5362736ef7ade892d01f85cc4d713e9f32a79fcb324452fb9076ccafc5c Oct 30 02:10:48.714158 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.714125 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:48.714235 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.714202 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:48.714273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:48.714232 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:49.404277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:49.404232 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" event={"ID":"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3","Type":"ContainerStarted","Data":"a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da"} Oct 30 02:10:49.404480 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:49.404301 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" event={"ID":"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3","Type":"ContainerStarted","Data":"32c7b5362736ef7ade892d01f85cc4d713e9f32a79fcb324452fb9076ccafc5c"} Oct 30 02:10:49.462290 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:49.462229 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" podStartSLOduration=1.4622150729999999 podStartE2EDuration="1.462215073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:49.461227201 +0000 UTC m=+2527.964159428" watchObservedRunningTime="2026-10-30 02:10:49.462215073 +0000 UTC m=+2527.965147293" Oct 30 02:10:50.391812 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:50.391761 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:50.393817 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:50.393763 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 02:10:50.393987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:50.393853 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 02:10:51.378553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:51.371001 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 02:10:51.378553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:51.371089 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 02:10:52.385554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:52.371993 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 02:10:52.385554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:52.372071 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 02:10:53.379554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:53.373996 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 02:10:53.379554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:53.374069 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 02:10:54.372334 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:54.372280 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 02:10:54.373034 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:54.372995 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 02:10:55.369914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:55.369867 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 02:10:55.370130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:55.369930 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 02:10:56.374605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:56.374570 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:56.375110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:56.375081 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:10:56.375803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:10:56.375782 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:11:17.678920 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:11:17.678856 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:11:19.005187 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:19.005143 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc"] Oct 30 02:11:24.005852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.005741 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" podUID="6cc33552-ba67-42dd-bedf-73513dda1dac" containerName="payload-pre-processing" containerID="cri-o://7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433" gracePeriod=25 Oct 30 02:11:24.308477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.308443 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:11:24.373274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.373232 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc33552-ba67-42dd-bedf-73513dda1dac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc33552-ba67-42dd-bedf-73513dda1dac-plugins-config-volume\") pod \"6cc33552-ba67-42dd-bedf-73513dda1dac\" (UID: \"6cc33552-ba67-42dd-bedf-73513dda1dac\") " Oct 30 02:11:24.373630 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.373599 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc33552-ba67-42dd-bedf-73513dda1dac-kube-api-access-98rk5\" (UniqueName: \"kubernetes.io/projected/6cc33552-ba67-42dd-bedf-73513dda1dac-kube-api-access-98rk5\") pod \"6cc33552-ba67-42dd-bedf-73513dda1dac\" (UID: \"6cc33552-ba67-42dd-bedf-73513dda1dac\") " Oct 30 02:11:24.374061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.374033 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc33552-ba67-42dd-bedf-73513dda1dac-plugins-config-volume" pod "6cc33552-ba67-42dd-bedf-73513dda1dac" (UID: "6cc33552-ba67-42dd-bedf-73513dda1dac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:24.374316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.374295 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc33552-ba67-42dd-bedf-73513dda1dac-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:11:24.376121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.376094 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc33552-ba67-42dd-bedf-73513dda1dac-kube-api-access-98rk5" pod "6cc33552-ba67-42dd-bedf-73513dda1dac" (UID: "6cc33552-ba67-42dd-bedf-73513dda1dac"). InnerVolumeSpecName "kube-api-access-98rk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:24.475903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.475855 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98rk5\" (UniqueName: \"kubernetes.io/projected/6cc33552-ba67-42dd-bedf-73513dda1dac-kube-api-access-98rk5\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:11:24.758885 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.758848 5035 generic.go:356] "Generic (PLEG): container finished" podID="6cc33552-ba67-42dd-bedf-73513dda1dac" containerID="7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433" exitCode=0 Oct 30 02:11:24.759081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.758895 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" Oct 30 02:11:24.759081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.758930 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" event={"ID":"6cc33552-ba67-42dd-bedf-73513dda1dac","Type":"ContainerDied","Data":"7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433"} Oct 30 02:11:24.759081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.758989 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc" event={"ID":"6cc33552-ba67-42dd-bedf-73513dda1dac","Type":"ContainerDied","Data":"ddbb50bba020e613b03922095bf3c28718eb20c9dae112a273c5cb9d642f43a3"} Oct 30 02:11:24.759081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.759011 5035 scope.go:122] "RemoveContainer" containerID="7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433" Oct 30 02:11:24.771081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.771055 5035 scope.go:122] "RemoveContainer" containerID="7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433" Oct 30 02:11:24.771456 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:11:24.771435 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433\": container with ID starting with 7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433 not found: ID does not exist" containerID="7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433" Oct 30 02:11:24.771510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.771465 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433"} err="failed to get container status \"7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433\": rpc error: code = NotFound desc = could not find container \"7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433\": container with ID starting with 7ac785e5f5b15bf7006abf3b350109f14ea6e36829f2144013d0e21c0f0bd433 not found: ID does not exist" Oct 30 02:11:24.819980 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.819935 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc"] Oct 30 02:11:24.834018 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:24.833976 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1028d-5747ccb875-vxckc"] Oct 30 02:11:25.645800 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:11:25.645760 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc33552-ba67-42dd-bedf-73513dda1dac" path="/var/lib/kubelet/pods/6cc33552-ba67-42dd-bedf-73513dda1dac/volumes" Oct 30 02:11:47.782594 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:11:47.782546 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:12:17.631259 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:12:17.631218 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:12:47.632259 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:12:47.632210 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:13:17.329199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:17.329109 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv"] Oct 30 02:13:17.329841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:17.329330 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" containerID="cri-o://b05fa652a5d39e8c4336cd073f4f60a218a1f7763164130d73c786021e6f46ec" gracePeriod=30 Oct 30 02:13:17.631195 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:13:17.631096 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:13:22.742761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.742723 5035 generic.go:356] "Generic (PLEG): container finished" podID="8aadb65e-4b07-40c9-8711-b99d41248123" containerID="b05fa652a5d39e8c4336cd073f4f60a218a1f7763164130d73c786021e6f46ec" exitCode=0 Oct 30 02:13:22.743172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.742794 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" event={"ID":"8aadb65e-4b07-40c9-8711-b99d41248123","Type":"ContainerDied","Data":"b05fa652a5d39e8c4336cd073f4f60a218a1f7763164130d73c786021e6f46ec"} Oct 30 02:13:22.778650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.778620 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 02:13:22.880245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880199 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-certs\") pod \"8aadb65e-4b07-40c9-8711-b99d41248123\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " Oct 30 02:13:22.880473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880338 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aadb65e-4b07-40c9-8711-b99d41248123-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aadb65e-4b07-40c9-8711-b99d41248123-istio-podinfo\") pod \"8aadb65e-4b07-40c9-8711-b99d41248123\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " Oct 30 02:13:22.880473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880412 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aadb65e-4b07-40c9-8711-b99d41248123-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aadb65e-4b07-40c9-8711-b99d41248123-istiod-ca-cert\") pod \"8aadb65e-4b07-40c9-8711-b99d41248123\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " Oct 30 02:13:22.880637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880572 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-certs" pod "8aadb65e-4b07-40c9-8711-b99d41248123" (UID: "8aadb65e-4b07-40c9-8711-b99d41248123"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:22.880637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880609 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-data\") pod \"8aadb65e-4b07-40c9-8711-b99d41248123\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " Oct 30 02:13:22.880748 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880652 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-credential-socket\") pod \"8aadb65e-4b07-40c9-8711-b99d41248123\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " Oct 30 02:13:22.880799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880759 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-kube-api-access-q2948\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-kube-api-access-q2948\") pod \"8aadb65e-4b07-40c9-8711-b99d41248123\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " Oct 30 02:13:22.880862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880813 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-socket\") pod \"8aadb65e-4b07-40c9-8711-b99d41248123\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " Oct 30 02:13:22.880862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880844 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-data" pod "8aadb65e-4b07-40c9-8711-b99d41248123" (UID: "8aadb65e-4b07-40c9-8711-b99d41248123"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:22.880963 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880859 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-istio-token\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-istio-token\") pod \"8aadb65e-4b07-40c9-8711-b99d41248123\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " Oct 30 02:13:22.880963 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880891 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-envoy\") pod \"8aadb65e-4b07-40c9-8711-b99d41248123\" (UID: \"8aadb65e-4b07-40c9-8711-b99d41248123\") " Oct 30 02:13:22.881057 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880957 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-credential-socket" pod "8aadb65e-4b07-40c9-8711-b99d41248123" (UID: "8aadb65e-4b07-40c9-8711-b99d41248123"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:22.881057 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.880982 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aadb65e-4b07-40c9-8711-b99d41248123-istiod-ca-cert" pod "8aadb65e-4b07-40c9-8711-b99d41248123" (UID: "8aadb65e-4b07-40c9-8711-b99d41248123"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:22.881157 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.881138 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-socket" pod "8aadb65e-4b07-40c9-8711-b99d41248123" (UID: "8aadb65e-4b07-40c9-8711-b99d41248123"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:22.881695 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.881668 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:22.881695 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.881697 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:22.881916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.881714 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:22.881916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.881730 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:22.881916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.881744 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aadb65e-4b07-40c9-8711-b99d41248123-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:22.883161 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.883132 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-kube-api-access-q2948" pod "8aadb65e-4b07-40c9-8711-b99d41248123" (UID: "8aadb65e-4b07-40c9-8711-b99d41248123"). InnerVolumeSpecName "kube-api-access-q2948". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:22.883161 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.883141 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-envoy" pod "8aadb65e-4b07-40c9-8711-b99d41248123" (UID: "8aadb65e-4b07-40c9-8711-b99d41248123"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:22.883344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.883144 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aadb65e-4b07-40c9-8711-b99d41248123-istio-podinfo" pod "8aadb65e-4b07-40c9-8711-b99d41248123" (UID: "8aadb65e-4b07-40c9-8711-b99d41248123"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:22.883344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.883176 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-istio-token" pod "8aadb65e-4b07-40c9-8711-b99d41248123" (UID: "8aadb65e-4b07-40c9-8711-b99d41248123"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:22.983103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.983059 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aadb65e-4b07-40c9-8711-b99d41248123-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:22.983103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.983100 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2948\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-kube-api-access-q2948\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:22.983378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.983115 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aadb65e-4b07-40c9-8711-b99d41248123-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:22.983378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:22.983130 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aadb65e-4b07-40c9-8711-b99d41248123-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:23.752481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:23.752448 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" Oct 30 02:13:23.752481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:23.752466 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv" event={"ID":"8aadb65e-4b07-40c9-8711-b99d41248123","Type":"ContainerDied","Data":"69ccaabb63f4212cfef15ff74493dbd5de22ee670d6ef338287ba1aa562e3374"} Oct 30 02:13:23.752999 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:23.752549 5035 scope.go:122] "RemoveContainer" containerID="b05fa652a5d39e8c4336cd073f4f60a218a1f7763164130d73c786021e6f46ec" Oct 30 02:13:23.825125 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:23.825087 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv"] Oct 30 02:13:23.838971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:23.838932 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-m8dlv"] Oct 30 02:13:25.644210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:25.644176 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" path="/var/lib/kubelet/pods/8aadb65e-4b07-40c9-8711-b99d41248123/volumes" Oct 30 02:13:29.628102 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:29.628062 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h"] Oct 30 02:13:29.628589 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:29.628324 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" containerID="cri-o://ab8ee71dd0097d79c0cbd6852124d7093c3b041546501018e4d122edf1a0e853" gracePeriod=30 Oct 30 02:13:34.846223 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:34.846182 5035 generic.go:356] "Generic (PLEG): container finished" podID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerID="ab8ee71dd0097d79c0cbd6852124d7093c3b041546501018e4d122edf1a0e853" exitCode=0 Oct 30 02:13:34.846692 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:34.846230 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" event={"ID":"b359b740-709a-4fd9-ade6-4bf08d7b827b","Type":"ContainerDied","Data":"ab8ee71dd0097d79c0cbd6852124d7093c3b041546501018e4d122edf1a0e853"} Oct 30 02:13:34.846692 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:34.846355 5035 scope.go:122] "RemoveContainer" containerID="a466a4de1b11b6eccf402360c9dc56a62dbc58114254ae93d9987a197c4627f3" Oct 30 02:13:34.931709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:34.931678 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 02:13:35.082029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.081986 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-kube-api-access-l2j49\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-kube-api-access-l2j49\") pod \"b359b740-709a-4fd9-ade6-4bf08d7b827b\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " Oct 30 02:13:35.082220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082057 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-data\") pod \"b359b740-709a-4fd9-ade6-4bf08d7b827b\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " Oct 30 02:13:35.082220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082116 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-socket\") pod \"b359b740-709a-4fd9-ade6-4bf08d7b827b\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " Oct 30 02:13:35.082220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082137 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-token\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-token\") pod \"b359b740-709a-4fd9-ade6-4bf08d7b827b\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " Oct 30 02:13:35.082220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082157 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-envoy\") pod \"b359b740-709a-4fd9-ade6-4bf08d7b827b\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " Oct 30 02:13:35.082220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082201 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-credential-socket\") pod \"b359b740-709a-4fd9-ade6-4bf08d7b827b\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " Oct 30 02:13:35.082507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082239 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-podinfo\") pod \"b359b740-709a-4fd9-ade6-4bf08d7b827b\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " Oct 30 02:13:35.082507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082313 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b359b740-709a-4fd9-ade6-4bf08d7b827b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b359b740-709a-4fd9-ade6-4bf08d7b827b-istiod-ca-cert\") pod \"b359b740-709a-4fd9-ade6-4bf08d7b827b\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " Oct 30 02:13:35.082507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082345 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-certs\") pod \"b359b740-709a-4fd9-ade6-4bf08d7b827b\" (UID: \"b359b740-709a-4fd9-ade6-4bf08d7b827b\") " Oct 30 02:13:35.082507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082379 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-data" pod "b359b740-709a-4fd9-ade6-4bf08d7b827b" (UID: "b359b740-709a-4fd9-ade6-4bf08d7b827b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.082862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082773 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b359b740-709a-4fd9-ade6-4bf08d7b827b-istiod-ca-cert" pod "b359b740-709a-4fd9-ade6-4bf08d7b827b" (UID: "b359b740-709a-4fd9-ade6-4bf08d7b827b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:35.082924 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082905 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-credential-socket" pod "b359b740-709a-4fd9-ade6-4bf08d7b827b" (UID: "b359b740-709a-4fd9-ade6-4bf08d7b827b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.082984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.082925 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-socket" pod "b359b740-709a-4fd9-ade6-4bf08d7b827b" (UID: "b359b740-709a-4fd9-ade6-4bf08d7b827b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.083045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.083033 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-certs" pod "b359b740-709a-4fd9-ade6-4bf08d7b827b" (UID: "b359b740-709a-4fd9-ade6-4bf08d7b827b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.083192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.083169 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.083266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.083197 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.083266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.083214 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b359b740-709a-4fd9-ade6-4bf08d7b827b-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.083266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.083228 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.083266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.083244 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.084595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.084561 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-token" pod "b359b740-709a-4fd9-ade6-4bf08d7b827b" (UID: "b359b740-709a-4fd9-ade6-4bf08d7b827b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:35.084738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.084714 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-podinfo" pod "b359b740-709a-4fd9-ade6-4bf08d7b827b" (UID: "b359b740-709a-4fd9-ade6-4bf08d7b827b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:35.084917 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.084893 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-envoy" pod "b359b740-709a-4fd9-ade6-4bf08d7b827b" (UID: "b359b740-709a-4fd9-ade6-4bf08d7b827b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.085116 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.085094 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-kube-api-access-l2j49" pod "b359b740-709a-4fd9-ade6-4bf08d7b827b" (UID: "b359b740-709a-4fd9-ade6-4bf08d7b827b"). InnerVolumeSpecName "kube-api-access-l2j49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:35.184632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.184489 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.184632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.184578 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.184632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.184588 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b359b740-709a-4fd9-ade6-4bf08d7b827b-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.184632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.184597 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2j49\" (UniqueName: \"kubernetes.io/projected/b359b740-709a-4fd9-ade6-4bf08d7b827b-kube-api-access-l2j49\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.854914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.854877 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" event={"ID":"b359b740-709a-4fd9-ade6-4bf08d7b827b","Type":"ContainerDied","Data":"f8051db5ff8d1a021279c9c5ddee618f0297afa7c372a44e65a9870e317dca02"} Oct 30 02:13:35.855371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.854925 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h" Oct 30 02:13:35.855371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.854950 5035 scope.go:122] "RemoveContainer" containerID="ab8ee71dd0097d79c0cbd6852124d7093c3b041546501018e4d122edf1a0e853" Oct 30 02:13:35.934916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.934805 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h"] Oct 30 02:13:35.952128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:35.952066 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8pw9h"] Oct 30 02:13:37.645602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:37.645566 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" path="/var/lib/kubelet/pods/b359b740-709a-4fd9-ade6-4bf08d7b827b/volumes" Oct 30 02:13:43.769514 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:43.769483 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:13:43.784760 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:43.784727 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:13:43.811935 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:43.811890 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:13:43.828450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:13:43.828415 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:13:47.630772 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:13:47.630728 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:14:00.531078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.531038 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9"] Oct 30 02:14:00.531813 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.531793 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" Oct 30 02:14:00.531813 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.531813 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" Oct 30 02:14:00.531976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.531829 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" Oct 30 02:14:00.531976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.531834 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" Oct 30 02:14:00.531976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.531854 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" Oct 30 02:14:00.531976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.531858 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" Oct 30 02:14:00.531976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.531879 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6cc33552-ba67-42dd-bedf-73513dda1dac" containerName="payload-pre-processing" Oct 30 02:14:00.531976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.531885 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc33552-ba67-42dd-bedf-73513dda1dac" containerName="payload-pre-processing" Oct 30 02:14:00.532291 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.532009 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" Oct 30 02:14:00.532291 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.532018 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b359b740-709a-4fd9-ade6-4bf08d7b827b" containerName="istio-proxy" Oct 30 02:14:00.532291 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.532028 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aadb65e-4b07-40c9-8711-b99d41248123" containerName="istio-proxy" Oct 30 02:14:00.532291 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.532036 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc33552-ba67-42dd-bedf-73513dda1dac" containerName="payload-pre-processing" Oct 30 02:14:00.537195 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.537159 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:00.556195 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.556158 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9"] Oct 30 02:14:00.609882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.609842 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-a0684f-dockercfg-znbbw\"" Oct 30 02:14:00.609882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.609885 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-a0684f\"" Oct 30 02:14:00.623552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.610934 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615904ed-4fd5-4af3-9cf3-490f849923df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9\" (UID: \"615904ed-4fd5-4af3-9cf3-490f849923df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:00.623552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.611142 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvvx\" (UniqueName: \"kubernetes.io/projected/615904ed-4fd5-4af3-9cf3-490f849923df-kube-api-access-stvvx\") pod \"payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9\" (UID: \"615904ed-4fd5-4af3-9cf3-490f849923df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:00.714605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.714557 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615904ed-4fd5-4af3-9cf3-490f849923df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9\" (UID: \"615904ed-4fd5-4af3-9cf3-490f849923df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:00.714916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.714893 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvvx\" (UniqueName: \"kubernetes.io/projected/615904ed-4fd5-4af3-9cf3-490f849923df-kube-api-access-stvvx\") pod \"payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9\" (UID: \"615904ed-4fd5-4af3-9cf3-490f849923df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:00.718335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.718294 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615904ed-4fd5-4af3-9cf3-490f849923df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9\" (UID: \"615904ed-4fd5-4af3-9cf3-490f849923df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:00.769736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.769704 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvvx\" (UniqueName: \"kubernetes.io/projected/615904ed-4fd5-4af3-9cf3-490f849923df-kube-api-access-stvvx\") pod \"payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9\" (UID: \"615904ed-4fd5-4af3-9cf3-490f849923df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:00.854158 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:00.854069 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:01.164159 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:01.164124 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9"] Oct 30 02:14:01.165518 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:14:01.165475 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615904ed_4fd5_4af3_9cf3_490f849923df.slice/crio-90a27a19e9942ec35c6fe34934fc25b953539de7e9363e580bcbdb3dabce9f96 WatchSource:0}: Error finding container 90a27a19e9942ec35c6fe34934fc25b953539de7e9363e580bcbdb3dabce9f96: Status 404 returned error can't find the container with id 90a27a19e9942ec35c6fe34934fc25b953539de7e9363e580bcbdb3dabce9f96 Oct 30 02:14:02.088890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:02.088837 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" event={"ID":"615904ed-4fd5-4af3-9cf3-490f849923df","Type":"ContainerStarted","Data":"be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f"} Oct 30 02:14:02.089313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:02.088910 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" event={"ID":"615904ed-4fd5-4af3-9cf3-490f849923df","Type":"ContainerStarted","Data":"90a27a19e9942ec35c6fe34934fc25b953539de7e9363e580bcbdb3dabce9f96"} Oct 30 02:14:02.089313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:02.088937 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:02.136639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:02.136577 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" podStartSLOduration=2.136561623 podStartE2EDuration="2.136561623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:02.132453662 +0000 UTC m=+2720.635385882" watchObservedRunningTime="2026-10-30 02:14:02.136561623 +0000 UTC m=+2720.639493839" Oct 30 02:14:06.282879 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:06.282840 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj"] Oct 30 02:14:06.283321 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:06.283065 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" containerID="cri-o://1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802" gracePeriod=30 Oct 30 02:14:11.345614 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:14:11.345565 5035 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19141fbc_b473_46d4_8383_658fdc21da18.slice/crio-1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:14:11.682387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.682352 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:14:11.744821 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.744770 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19141fbc-b473-46d4-8383-658fdc21da18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19141fbc-b473-46d4-8383-658fdc21da18-istio-podinfo\") pod \"19141fbc-b473-46d4-8383-658fdc21da18\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " Oct 30 02:14:11.745021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.744900 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19141fbc-b473-46d4-8383-658fdc21da18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19141fbc-b473-46d4-8383-658fdc21da18-istiod-ca-cert\") pod \"19141fbc-b473-46d4-8383-658fdc21da18\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " Oct 30 02:14:11.745021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.744967 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-envoy\") pod \"19141fbc-b473-46d4-8383-658fdc21da18\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " Oct 30 02:14:11.745021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.745007 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-certs\") pod \"19141fbc-b473-46d4-8383-658fdc21da18\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " Oct 30 02:14:11.745129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.745041 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-socket\") pod \"19141fbc-b473-46d4-8383-658fdc21da18\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " Oct 30 02:14:11.745170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.745145 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-data\") pod \"19141fbc-b473-46d4-8383-658fdc21da18\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " Oct 30 02:14:11.745305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.745279 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-istio-token\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-istio-token\") pod \"19141fbc-b473-46d4-8383-658fdc21da18\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " Oct 30 02:14:11.745453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.745386 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-kube-api-access-hp2t7\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-kube-api-access-hp2t7\") pod \"19141fbc-b473-46d4-8383-658fdc21da18\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " Oct 30 02:14:11.745544 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.745460 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-credential-socket\") pod \"19141fbc-b473-46d4-8383-658fdc21da18\" (UID: \"19141fbc-b473-46d4-8383-658fdc21da18\") " Oct 30 02:14:11.748061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.747146 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-socket" pod "19141fbc-b473-46d4-8383-658fdc21da18" (UID: "19141fbc-b473-46d4-8383-658fdc21da18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.748061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.747356 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19141fbc-b473-46d4-8383-658fdc21da18-istiod-ca-cert" pod "19141fbc-b473-46d4-8383-658fdc21da18" (UID: "19141fbc-b473-46d4-8383-658fdc21da18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:11.748061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.747392 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-certs" pod "19141fbc-b473-46d4-8383-658fdc21da18" (UID: "19141fbc-b473-46d4-8383-658fdc21da18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.748061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.747697 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-data" pod "19141fbc-b473-46d4-8383-658fdc21da18" (UID: "19141fbc-b473-46d4-8383-658fdc21da18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.748061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.747756 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-credential-socket" pod "19141fbc-b473-46d4-8383-658fdc21da18" (UID: "19141fbc-b473-46d4-8383-658fdc21da18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.748061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.747943 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-envoy" pod "19141fbc-b473-46d4-8383-658fdc21da18" (UID: "19141fbc-b473-46d4-8383-658fdc21da18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:11.749994 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.749970 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19141fbc-b473-46d4-8383-658fdc21da18-istio-podinfo" pod "19141fbc-b473-46d4-8383-658fdc21da18" (UID: "19141fbc-b473-46d4-8383-658fdc21da18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:11.809560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.808922 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-kube-api-access-hp2t7" pod "19141fbc-b473-46d4-8383-658fdc21da18" (UID: "19141fbc-b473-46d4-8383-658fdc21da18"). InnerVolumeSpecName "kube-api-access-hp2t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:11.848553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.847791 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp2t7\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-kube-api-access-hp2t7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.848553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.847831 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.848553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.847858 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19141fbc-b473-46d4-8383-658fdc21da18-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.848553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.847872 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19141fbc-b473-46d4-8383-658fdc21da18-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.848553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.847888 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.848553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.847901 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.848553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.847914 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.848553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.847949 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19141fbc-b473-46d4-8383-658fdc21da18-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:11.889888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.889798 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-istio-token" pod "19141fbc-b473-46d4-8383-658fdc21da18" (UID: "19141fbc-b473-46d4-8383-658fdc21da18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:11.949208 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:11.949089 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19141fbc-b473-46d4-8383-658fdc21da18-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:12.187021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:12.186979 5035 generic.go:356] "Generic (PLEG): container finished" podID="19141fbc-b473-46d4-8383-658fdc21da18" containerID="1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802" exitCode=0 Oct 30 02:14:12.187247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:12.187036 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" Oct 30 02:14:12.187307 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:12.187253 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" event={"ID":"19141fbc-b473-46d4-8383-658fdc21da18","Type":"ContainerDied","Data":"1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802"} Oct 30 02:14:12.187307 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:12.187296 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj" event={"ID":"19141fbc-b473-46d4-8383-658fdc21da18","Type":"ContainerDied","Data":"d80ef24bf8b6b31fe8ec6feb59fc252d36122292e355d10483bbb46fafd121e7"} Oct 30 02:14:12.187483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:12.187329 5035 scope.go:122] "RemoveContainer" containerID="1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802" Oct 30 02:14:12.204212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:12.204188 5035 scope.go:122] "RemoveContainer" containerID="1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802" Oct 30 02:14:12.204564 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:14:12.204521 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802\": container with ID starting with 1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802 not found: ID does not exist" containerID="1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802" Oct 30 02:14:12.204665 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:12.204577 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802"} err="failed to get container status \"1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802\": rpc error: code = NotFound desc = could not find container \"1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802\": container with ID starting with 1b970d3649868779c47d0cfa7118dd0acd34bf5ace76ab28f61eefadaa69a802 not found: ID does not exist" Oct 30 02:14:12.247362 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:12.247322 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj"] Oct 30 02:14:12.260802 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:12.260764 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-jfwzj"] Oct 30 02:14:13.099948 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:13.099902 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:14:13.645059 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:13.645022 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19141fbc-b473-46d4-8383-658fdc21da18" path="/var/lib/kubelet/pods/19141fbc-b473-46d4-8383-658fdc21da18/volumes" Oct 30 02:14:14.744816 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:14.744773 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh"] Oct 30 02:14:14.745276 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:14.745015 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" containerID="cri-o://839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b" gracePeriod=30 Oct 30 02:14:14.759028 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:14.758987 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9"] Oct 30 02:14:14.759236 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:14.759213 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" containerID="cri-o://aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d" gracePeriod=30 Oct 30 02:14:17.630795 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:14:17.630754 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:14:19.558279 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:19.558237 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:14:19.558721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:19.558321 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:14:20.001564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:19.997932 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv"] Oct 30 02:14:20.001564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:19.999138 5035 cpu_manager.go:414] "containerMap: removing container" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" Oct 30 02:14:20.001564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:19.999164 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" Oct 30 02:14:20.001564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:19.999350 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="19141fbc-b473-46d4-8383-658fdc21da18" containerName="istio-proxy" Oct 30 02:14:20.011289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.011173 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv"] Oct 30 02:14:20.026764 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.026661 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:20.046667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.046387 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-93ec46\"" Oct 30 02:14:20.047486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.047444 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-93ec46-dockercfg-x58r8\"" Oct 30 02:14:20.153644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.153597 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0e8297-c571-4105-b69a-c7eb9c3938be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv\" (UID: \"6c0e8297-c571-4105-b69a-c7eb9c3938be\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:20.153997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.153966 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284l8\" (UniqueName: \"kubernetes.io/projected/6c0e8297-c571-4105-b69a-c7eb9c3938be-kube-api-access-284l8\") pod \"payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv\" (UID: \"6c0e8297-c571-4105-b69a-c7eb9c3938be\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:20.249679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.249651 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:14:20.254072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.254044 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 02:14:20.255315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.255286 5035 generic.go:356] "Generic (PLEG): container finished" podID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerID="aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d" exitCode=0 Oct 30 02:14:20.255450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.255361 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" event={"ID":"41a38459-65bc-418d-93e9-13f88b7f9c7d","Type":"ContainerDied","Data":"aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d"} Oct 30 02:14:20.255450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.255379 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0e8297-c571-4105-b69a-c7eb9c3938be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv\" (UID: \"6c0e8297-c571-4105-b69a-c7eb9c3938be\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:20.255450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.255426 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" event={"ID":"41a38459-65bc-418d-93e9-13f88b7f9c7d","Type":"ContainerDied","Data":"d0de6d70d0f1dd9925707461bd40249c9d842baf58d5e5da0b365a650033c2b7"} Oct 30 02:14:20.255660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.255466 5035 scope.go:122] "RemoveContainer" containerID="aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d" Oct 30 02:14:20.255660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.255470 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284l8\" (UniqueName: \"kubernetes.io/projected/6c0e8297-c571-4105-b69a-c7eb9c3938be-kube-api-access-284l8\") pod \"payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv\" (UID: \"6c0e8297-c571-4105-b69a-c7eb9c3938be\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:20.256310 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.256282 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0e8297-c571-4105-b69a-c7eb9c3938be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv\" (UID: \"6c0e8297-c571-4105-b69a-c7eb9c3938be\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:20.257136 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.257116 5035 generic.go:356] "Generic (PLEG): container finished" podID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerID="839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b" exitCode=0 Oct 30 02:14:20.257210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.257167 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" Oct 30 02:14:20.257210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.257199 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" event={"ID":"aae44149-6c30-41ae-bfc1-8c07dbc642a0","Type":"ContainerDied","Data":"839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b"} Oct 30 02:14:20.257299 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.257245 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh" event={"ID":"aae44149-6c30-41ae-bfc1-8c07dbc642a0","Type":"ContainerDied","Data":"12d08e47e99cd8340ef3502cecd7f5d6283f5a7451027dbcf0f56ff43b59ca4e"} Oct 30 02:14:20.267197 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.267172 5035 scope.go:122] "RemoveContainer" containerID="aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d" Oct 30 02:14:20.267544 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:14:20.267505 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d\": container with ID starting with aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d not found: ID does not exist" containerID="aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d" Oct 30 02:14:20.267636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.267558 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d"} err="failed to get container status \"aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d\": rpc error: code = NotFound desc = could not find container \"aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d\": container with ID starting with aabbc29e6e631ad2307df0e671a486fef4df7edd198cd7d51855e80c22abe05d not found: ID does not exist" Oct 30 02:14:20.267636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.267583 5035 scope.go:122] "RemoveContainer" containerID="839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b" Oct 30 02:14:20.278639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.278620 5035 scope.go:122] "RemoveContainer" containerID="839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b" Oct 30 02:14:20.278972 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:14:20.278947 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b\": container with ID starting with 839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b not found: ID does not exist" containerID="839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b" Oct 30 02:14:20.279039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.278981 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b"} err="failed to get container status \"839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b\": rpc error: code = NotFound desc = could not find container \"839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b\": container with ID starting with 839008906c411f703cfa52f7a827576020c1e0417373fe819d59034020e3092b not found: ID does not exist" Oct 30 02:14:20.280463 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.280442 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284l8\" (UniqueName: \"kubernetes.io/projected/6c0e8297-c571-4105-b69a-c7eb9c3938be-kube-api-access-284l8\") pod \"payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv\" (UID: \"6c0e8297-c571-4105-b69a-c7eb9c3938be\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:20.356325 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356235 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-podinfo\") pod \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " Oct 30 02:14:20.356509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356335 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-certs\") pod \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " Oct 30 02:14:20.356509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356389 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-credential-socket\") pod \"41a38459-65bc-418d-93e9-13f88b7f9c7d\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " Oct 30 02:14:20.356509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356466 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-socket\") pod \"41a38459-65bc-418d-93e9-13f88b7f9c7d\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " Oct 30 02:14:20.356509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356497 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-envoy\") pod \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " Oct 30 02:14:20.356735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356537 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-kube-api-access-qdsbc\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-kube-api-access-qdsbc\") pod \"41a38459-65bc-418d-93e9-13f88b7f9c7d\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " Oct 30 02:14:20.356735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356564 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-envoy\") pod \"41a38459-65bc-418d-93e9-13f88b7f9c7d\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " Oct 30 02:14:20.356735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356590 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-data\") pod \"41a38459-65bc-418d-93e9-13f88b7f9c7d\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " Oct 30 02:14:20.356902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356740 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istiod-ca-cert\") pod \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " Oct 30 02:14:20.356902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356766 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-credential-socket" pod "41a38459-65bc-418d-93e9-13f88b7f9c7d" (UID: "41a38459-65bc-418d-93e9-13f88b7f9c7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.356902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356812 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-token\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-token\") pod \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " Oct 30 02:14:20.356902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356854 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a38459-65bc-418d-93e9-13f88b7f9c7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a38459-65bc-418d-93e9-13f88b7f9c7d-istiod-ca-cert\") pod \"41a38459-65bc-418d-93e9-13f88b7f9c7d\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " Oct 30 02:14:20.356902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356885 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-credential-socket\") pod \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " Oct 30 02:14:20.357165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356913 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-certs\") pod \"41a38459-65bc-418d-93e9-13f88b7f9c7d\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " Oct 30 02:14:20.357165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.356959 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-podinfo\") pod \"41a38459-65bc-418d-93e9-13f88b7f9c7d\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " Oct 30 02:14:20.357165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.357018 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-socket" pod "41a38459-65bc-418d-93e9-13f88b7f9c7d" (UID: "41a38459-65bc-418d-93e9-13f88b7f9c7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.357165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.357022 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-socket\") pod \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " Oct 30 02:14:20.357165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.357135 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-kube-api-access-2hhv6\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-kube-api-access-2hhv6\") pod \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " Oct 30 02:14:20.357433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.357191 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-socket" pod "aae44149-6c30-41ae-bfc1-8c07dbc642a0" (UID: "aae44149-6c30-41ae-bfc1-8c07dbc642a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.357433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.357213 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-data\") pod \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\" (UID: \"aae44149-6c30-41ae-bfc1-8c07dbc642a0\") " Oct 30 02:14:20.357433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.357254 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-token\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-token\") pod \"41a38459-65bc-418d-93e9-13f88b7f9c7d\" (UID: \"41a38459-65bc-418d-93e9-13f88b7f9c7d\") " Oct 30 02:14:20.357433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.357268 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-certs" pod "aae44149-6c30-41ae-bfc1-8c07dbc642a0" (UID: "aae44149-6c30-41ae-bfc1-8c07dbc642a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.358083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.358039 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.358083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.358071 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.358083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.358086 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.358399 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.358101 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.358931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.358898 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a38459-65bc-418d-93e9-13f88b7f9c7d-istiod-ca-cert" pod "41a38459-65bc-418d-93e9-13f88b7f9c7d" (UID: "41a38459-65bc-418d-93e9-13f88b7f9c7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:20.359221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.359184 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-data" pod "aae44149-6c30-41ae-bfc1-8c07dbc642a0" (UID: "aae44149-6c30-41ae-bfc1-8c07dbc642a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.359768 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.359383 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-certs" pod "41a38459-65bc-418d-93e9-13f88b7f9c7d" (UID: "41a38459-65bc-418d-93e9-13f88b7f9c7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.359768 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.359584 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-data" pod "41a38459-65bc-418d-93e9-13f88b7f9c7d" (UID: "41a38459-65bc-418d-93e9-13f88b7f9c7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.359965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.359782 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-credential-socket" pod "aae44149-6c30-41ae-bfc1-8c07dbc642a0" (UID: "aae44149-6c30-41ae-bfc1-8c07dbc642a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.360225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.360183 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istiod-ca-cert" pod "aae44149-6c30-41ae-bfc1-8c07dbc642a0" (UID: "aae44149-6c30-41ae-bfc1-8c07dbc642a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:20.361164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.361136 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-envoy" pod "41a38459-65bc-418d-93e9-13f88b7f9c7d" (UID: "41a38459-65bc-418d-93e9-13f88b7f9c7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.361463 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.361435 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-kube-api-access-qdsbc" pod "41a38459-65bc-418d-93e9-13f88b7f9c7d" (UID: "41a38459-65bc-418d-93e9-13f88b7f9c7d"). InnerVolumeSpecName "kube-api-access-qdsbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:20.362083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.362057 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-token" pod "aae44149-6c30-41ae-bfc1-8c07dbc642a0" (UID: "aae44149-6c30-41ae-bfc1-8c07dbc642a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:20.362887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.362862 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-podinfo" pod "41a38459-65bc-418d-93e9-13f88b7f9c7d" (UID: "41a38459-65bc-418d-93e9-13f88b7f9c7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:20.362981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.362878 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-podinfo" pod "aae44149-6c30-41ae-bfc1-8c07dbc642a0" (UID: "aae44149-6c30-41ae-bfc1-8c07dbc642a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:20.362981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.362930 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-kube-api-access-2hhv6" pod "aae44149-6c30-41ae-bfc1-8c07dbc642a0" (UID: "aae44149-6c30-41ae-bfc1-8c07dbc642a0"). InnerVolumeSpecName "kube-api-access-2hhv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:20.363077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.362992 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-envoy" pod "aae44149-6c30-41ae-bfc1-8c07dbc642a0" (UID: "aae44149-6c30-41ae-bfc1-8c07dbc642a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:20.363077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.363015 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-token" pod "41a38459-65bc-418d-93e9-13f88b7f9c7d" (UID: "41a38459-65bc-418d-93e9-13f88b7f9c7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:20.384192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.384151 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.459927 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.459967 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.459982 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a38459-65bc-418d-93e9-13f88b7f9c7d-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.459995 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460010 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460025 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460038 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hhv6\" (UniqueName: \"kubernetes.io/projected/aae44149-6c30-41ae-bfc1-8c07dbc642a0-kube-api-access-2hhv6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460052 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460064 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460075 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460085 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae44149-6c30-41ae-bfc1-8c07dbc642a0-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460099 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdsbc\" (UniqueName: \"kubernetes.io/projected/41a38459-65bc-418d-93e9-13f88b7f9c7d-kube-api-access-qdsbc\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460113 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.462207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.460126 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a38459-65bc-418d-93e9-13f88b7f9c7d-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:14:20.613114 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.611481 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh"] Oct 30 02:14:20.619878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.619832 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-s2zsh"] Oct 30 02:14:20.703594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:20.703546 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv"] Oct 30 02:14:20.706378 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:14:20.706341 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0e8297_c571_4105_b69a_c7eb9c3938be.slice/crio-4de9588002fcc3bb56d2ee1021ace3fc4b0ca72897eda40b2cfb580a468e876b WatchSource:0}: Error finding container 4de9588002fcc3bb56d2ee1021ace3fc4b0ca72897eda40b2cfb580a468e876b: Status 404 returned error can't find the container with id 4de9588002fcc3bb56d2ee1021ace3fc4b0ca72897eda40b2cfb580a468e876b Oct 30 02:14:21.265990 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:21.265954 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9" Oct 30 02:14:21.268081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:21.268046 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" event={"ID":"6c0e8297-c571-4105-b69a-c7eb9c3938be","Type":"ContainerStarted","Data":"d1af9cc72a9a0255020e279ffaa2cdd2dbabf4835950a9d676e310431ce30975"} Oct 30 02:14:21.268257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:21.268114 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:21.268257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:21.268137 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" event={"ID":"6c0e8297-c571-4105-b69a-c7eb9c3938be","Type":"ContainerStarted","Data":"4de9588002fcc3bb56d2ee1021ace3fc4b0ca72897eda40b2cfb580a468e876b"} Oct 30 02:14:21.327966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:21.327897 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" podStartSLOduration=2.327882164 podStartE2EDuration="2.327882164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:21.327180806 +0000 UTC m=+2739.830113027" watchObservedRunningTime="2026-10-30 02:14:21.327882164 +0000 UTC m=+2739.830814382" Oct 30 02:14:21.369002 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:21.368959 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9"] Oct 30 02:14:21.376304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:21.376265 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-fxmw9"] Oct 30 02:14:21.653514 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:21.653422 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" path="/var/lib/kubelet/pods/41a38459-65bc-418d-93e9-13f88b7f9c7d/volumes" Oct 30 02:14:21.654232 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:21.654208 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" path="/var/lib/kubelet/pods/aae44149-6c30-41ae-bfc1-8c07dbc642a0/volumes" Oct 30 02:14:24.148074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.147984 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt"] Oct 30 02:14:24.148853 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.148835 5035 cpu_manager.go:414] "containerMap: removing container" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" Oct 30 02:14:24.148853 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.148854 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" Oct 30 02:14:24.149008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.148872 5035 cpu_manager.go:414] "containerMap: removing container" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" Oct 30 02:14:24.149008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.148880 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" Oct 30 02:14:24.149104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.149018 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae44149-6c30-41ae-bfc1-8c07dbc642a0" containerName="istio-proxy" Oct 30 02:14:24.149104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.149029 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a38459-65bc-418d-93e9-13f88b7f9c7d" containerName="istio-proxy" Oct 30 02:14:24.154357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.154330 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.177631 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.177591 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6b8bfa-serving-cert\"" Oct 30 02:14:24.184192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.184150 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt"] Oct 30 02:14:24.317298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.317250 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.317757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.317726 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4a780-b282-43f7-9173-d6bf26315614-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.317978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.317960 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-metrics-certs\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.318184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.318164 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctdn\" (UniqueName: \"kubernetes.io/projected/2bb4a780-b282-43f7-9173-d6bf26315614-kube-api-access-7ctdn\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.419975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.419865 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.419975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.419938 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4a780-b282-43f7-9173-d6bf26315614-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.419975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.419972 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-metrics-certs\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.420300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.420026 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctdn\" (UniqueName: \"kubernetes.io/projected/2bb4a780-b282-43f7-9173-d6bf26315614-kube-api-access-7ctdn\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.420932 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.420901 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4a780-b282-43f7-9173-d6bf26315614-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.423083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.423055 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-metrics-certs\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.423083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.423071 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.445030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.444997 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctdn\" (UniqueName: \"kubernetes.io/projected/2bb4a780-b282-43f7-9173-d6bf26315614-kube-api-access-7ctdn\") pod \"maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.483120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.483077 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:24.797848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.797812 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt"] Oct 30 02:14:24.798414 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:14:24.798386 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb4a780_b282_43f7_9173_d6bf26315614.slice/crio-9a68abddd615544d6eb0f53a8bb4684d8b23d61063984579d75f6ace1ceab9e9 WatchSource:0}: Error finding container 9a68abddd615544d6eb0f53a8bb4684d8b23d61063984579d75f6ace1ceab9e9: Status 404 returned error can't find the container with id 9a68abddd615544d6eb0f53a8bb4684d8b23d61063984579d75f6ace1ceab9e9 Oct 30 02:14:24.799976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:24.799954 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:14:25.318866 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:25.318813 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" event={"ID":"2bb4a780-b282-43f7-9173-d6bf26315614","Type":"ContainerStarted","Data":"410a259f6f5067e0ad30c7b043e6b4d97b43eef4282067730e45eada8eca133b"} Oct 30 02:14:25.319448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:25.318884 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" event={"ID":"2bb4a780-b282-43f7-9173-d6bf26315614","Type":"ContainerStarted","Data":"9a68abddd615544d6eb0f53a8bb4684d8b23d61063984579d75f6ace1ceab9e9"} Oct 30 02:14:25.319448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:25.318925 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:25.369939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:25.369875 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" podStartSLOduration=1.022451589 podStartE2EDuration="1.36985621s" totalImagesPullingTime="347.404621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:25.364677645 +0000 UTC m=+2743.867609878" watchObservedRunningTime="2026-10-30 02:14:25.36985621 +0000 UTC m=+2743.872788503" Oct 30 02:14:27.367136 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.367095 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9"] Oct 30 02:14:27.372081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.372050 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.399762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.399726 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-a0684f-serving-cert\"" Oct 30 02:14:27.404099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.404063 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9"] Oct 30 02:14:27.472818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.472777 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j892t\" (UniqueName: \"kubernetes.io/projected/8e72d15b-0f2c-4bbe-9dea-add735e9b124-kube-api-access-j892t\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.473002 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.472954 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.473103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.473087 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.473234 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.473209 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e72d15b-0f2c-4bbe-9dea-add735e9b124-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.574378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.574332 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.574639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.574454 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e72d15b-0f2c-4bbe-9dea-add735e9b124-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.574639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.574601 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j892t\" (UniqueName: \"kubernetes.io/projected/8e72d15b-0f2c-4bbe-9dea-add735e9b124-kube-api-access-j892t\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.574751 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.574699 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.575248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.575224 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e72d15b-0f2c-4bbe-9dea-add735e9b124-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.577197 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.577172 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.577299 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.577240 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.600506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.600460 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j892t\" (UniqueName: \"kubernetes.io/projected/8e72d15b-0f2c-4bbe-9dea-add735e9b124-kube-api-access-j892t\") pod \"maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:27.690219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:27.690120 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:28.008292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:28.008260 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9"] Oct 30 02:14:28.009630 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:14:28.009600 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e72d15b_0f2c_4bbe_9dea_add735e9b124.slice/crio-1c94a3ed31a28e13a00335ee264743d5245c7df9b77c923050be48fb0cfd7669 WatchSource:0}: Error finding container 1c94a3ed31a28e13a00335ee264743d5245c7df9b77c923050be48fb0cfd7669: Status 404 returned error can't find the container with id 1c94a3ed31a28e13a00335ee264743d5245c7df9b77c923050be48fb0cfd7669 Oct 30 02:14:28.343055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:28.343016 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" event={"ID":"8e72d15b-0f2c-4bbe-9dea-add735e9b124","Type":"ContainerStarted","Data":"1c94a3ed31a28e13a00335ee264743d5245c7df9b77c923050be48fb0cfd7669"} Oct 30 02:14:29.352605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:29.352564 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" event={"ID":"8e72d15b-0f2c-4bbe-9dea-add735e9b124","Type":"ContainerStarted","Data":"1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba"} Oct 30 02:14:29.353044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:29.352756 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:29.407486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:29.407427 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" podStartSLOduration=2.129888587 podStartE2EDuration="2.407412773s" totalImagesPullingTime="277.524186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:29.406433354 +0000 UTC m=+2747.909365598" watchObservedRunningTime="2026-10-30 02:14:29.407412773 +0000 UTC m=+2747.910344992" Oct 30 02:14:31.351016 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:31.350962 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:14:32.281099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:32.281057 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:14:35.366505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:35.366469 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:14:47.720116 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:14:47.720068 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:14:59.897850 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:59.897811 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r"] Oct 30 02:14:59.898376 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:14:59.898037 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" containerID="cri-o://e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a" gracePeriod=30 Oct 30 02:15:00.185194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.185096 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw"] Oct 30 02:15:00.192589 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.192555 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:00.205934 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.205902 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb"] Oct 30 02:15:00.231776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.231745 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:00.238205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.238172 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm"] Oct 30 02:15:00.245908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.245859 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6"] Oct 30 02:15:00.247801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.247763 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:00.256295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.256264 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw"] Oct 30 02:15:00.256676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.256654 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:00.259573 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.259487 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb"] Oct 30 02:15:00.265012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.264977 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4"] Oct 30 02:15:00.273023 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.272488 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:00.277308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.277281 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm"] Oct 30 02:15:00.284655 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.284619 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr"] Oct 30 02:15:00.290409 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.290379 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6"] Oct 30 02:15:00.290625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.290486 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:00.298075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.297994 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:15:00.305104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.305068 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbhn\" (UniqueName: \"kubernetes.io/projected/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-kube-api-access-nmbhn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb\" (UID: \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:00.305431 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.305410 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/459172b3-dfd3-46c0-9338-7640a59b6d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw\" (UID: \"459172b3-dfd3-46c0-9338-7640a59b6d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:00.305678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.305661 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8f3eda-f86d-4b97-adb7-131d606bf442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4\" (UID: \"ad8f3eda-f86d-4b97-adb7-131d606bf442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:00.305819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.305801 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgqs\" (UniqueName: \"kubernetes.io/projected/459172b3-dfd3-46c0-9338-7640a59b6d44-kube-api-access-pfgqs\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw\" (UID: \"459172b3-dfd3-46c0-9338-7640a59b6d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:00.306128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.306102 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6\" (UID: \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:00.306366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.306209 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/ad8f3eda-f86d-4b97-adb7-131d606bf442-kube-api-access-7slnv\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4\" (UID: \"ad8f3eda-f86d-4b97-adb7-131d606bf442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:00.306659 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.306439 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb\" (UID: \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:00.306935 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.306753 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2l8\" (UniqueName: \"kubernetes.io/projected/99be1c56-597c-4776-b219-44b73eaebfae-kube-api-access-jx2l8\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm\" (UID: \"99be1c56-597c-4776-b219-44b73eaebfae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:00.307010 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.306968 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be1c56-597c-4776-b219-44b73eaebfae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm\" (UID: \"99be1c56-597c-4776-b219-44b73eaebfae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:00.307255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.307029 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplsk\" (UniqueName: \"kubernetes.io/projected/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-kube-api-access-gplsk\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6\" (UID: \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:00.328203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.328168 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4"] Oct 30 02:15:00.334682 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.334646 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr"] Oct 30 02:15:00.414500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.414457 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/ad8f3eda-f86d-4b97-adb7-131d606bf442-kube-api-access-7slnv\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4\" (UID: \"ad8f3eda-f86d-4b97-adb7-131d606bf442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:00.414753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.414568 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb\" (UID: \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:00.414753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.414671 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hjk\" (UniqueName: \"kubernetes.io/projected/1884eb8f-8045-44a4-908c-13bbc9995d64-kube-api-access-67hjk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr\" (UID: \"1884eb8f-8045-44a4-908c-13bbc9995d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:00.414753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.414706 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2l8\" (UniqueName: \"kubernetes.io/projected/99be1c56-597c-4776-b219-44b73eaebfae-kube-api-access-jx2l8\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm\" (UID: \"99be1c56-597c-4776-b219-44b73eaebfae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:00.414753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.414736 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be1c56-597c-4776-b219-44b73eaebfae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm\" (UID: \"99be1c56-597c-4776-b219-44b73eaebfae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:00.415061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.414865 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplsk\" (UniqueName: \"kubernetes.io/projected/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-kube-api-access-gplsk\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6\" (UID: \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:00.415061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.414914 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1884eb8f-8045-44a4-908c-13bbc9995d64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr\" (UID: \"1884eb8f-8045-44a4-908c-13bbc9995d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:00.415061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.415008 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbhn\" (UniqueName: \"kubernetes.io/projected/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-kube-api-access-nmbhn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb\" (UID: \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:00.415061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.415032 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/459172b3-dfd3-46c0-9338-7640a59b6d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw\" (UID: \"459172b3-dfd3-46c0-9338-7640a59b6d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:00.415061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.415057 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8f3eda-f86d-4b97-adb7-131d606bf442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4\" (UID: \"ad8f3eda-f86d-4b97-adb7-131d606bf442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:00.415356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.415102 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgqs\" (UniqueName: \"kubernetes.io/projected/459172b3-dfd3-46c0-9338-7640a59b6d44-kube-api-access-pfgqs\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw\" (UID: \"459172b3-dfd3-46c0-9338-7640a59b6d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:00.415356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.415199 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6\" (UID: \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:00.415471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.415413 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb\" (UID: \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:00.416159 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.416133 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6\" (UID: \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:00.416322 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.416293 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8f3eda-f86d-4b97-adb7-131d606bf442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4\" (UID: \"ad8f3eda-f86d-4b97-adb7-131d606bf442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:00.416428 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.416292 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/459172b3-dfd3-46c0-9338-7640a59b6d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw\" (UID: \"459172b3-dfd3-46c0-9338-7640a59b6d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:00.418438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.418409 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be1c56-597c-4776-b219-44b73eaebfae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm\" (UID: \"99be1c56-597c-4776-b219-44b73eaebfae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:00.443620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.443353 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2l8\" (UniqueName: \"kubernetes.io/projected/99be1c56-597c-4776-b219-44b73eaebfae-kube-api-access-jx2l8\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm\" (UID: \"99be1c56-597c-4776-b219-44b73eaebfae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:00.443620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.443387 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgqs\" (UniqueName: \"kubernetes.io/projected/459172b3-dfd3-46c0-9338-7640a59b6d44-kube-api-access-pfgqs\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw\" (UID: \"459172b3-dfd3-46c0-9338-7640a59b6d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:00.443620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.443405 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbhn\" (UniqueName: \"kubernetes.io/projected/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-kube-api-access-nmbhn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb\" (UID: \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:00.443620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.443478 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/ad8f3eda-f86d-4b97-adb7-131d606bf442-kube-api-access-7slnv\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4\" (UID: \"ad8f3eda-f86d-4b97-adb7-131d606bf442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:00.443620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.443570 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplsk\" (UniqueName: \"kubernetes.io/projected/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-kube-api-access-gplsk\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6\" (UID: \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:00.516586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.516518 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hjk\" (UniqueName: \"kubernetes.io/projected/1884eb8f-8045-44a4-908c-13bbc9995d64-kube-api-access-67hjk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr\" (UID: \"1884eb8f-8045-44a4-908c-13bbc9995d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:00.516811 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.516624 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1884eb8f-8045-44a4-908c-13bbc9995d64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr\" (UID: \"1884eb8f-8045-44a4-908c-13bbc9995d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:00.517298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.517275 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1884eb8f-8045-44a4-908c-13bbc9995d64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr\" (UID: \"1884eb8f-8045-44a4-908c-13bbc9995d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:00.543243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.543193 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hjk\" (UniqueName: \"kubernetes.io/projected/1884eb8f-8045-44a4-908c-13bbc9995d64-kube-api-access-67hjk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr\" (UID: \"1884eb8f-8045-44a4-908c-13bbc9995d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:00.543446 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.543433 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:00.565016 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.564975 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:00.588812 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.588774 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:00.614346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.614305 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:00.629153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.628857 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:00.633090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.632515 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:00.868297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:00.868251 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw"] Oct 30 02:15:00.869423 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:00.869372 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459172b3_dfd3_46c0_9338_7640a59b6d44.slice/crio-1e6c40a4e92dfe790609a2a9082d61387fa21a6bd7fb61a753f3be92f3efa60b WatchSource:0}: Error finding container 1e6c40a4e92dfe790609a2a9082d61387fa21a6bd7fb61a753f3be92f3efa60b: Status 404 returned error can't find the container with id 1e6c40a4e92dfe790609a2a9082d61387fa21a6bd7fb61a753f3be92f3efa60b Oct 30 02:15:01.018497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.018463 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6"] Oct 30 02:15:01.038590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.038556 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4"] Oct 30 02:15:01.070413 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:01.070368 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea2f6d5_3cd2_4d6f_806e_de5e3f560319.slice/crio-10cf2a3ecaae009a3a198e0ec8d3b334a786503fd3d645dd404eb63baee7b479 WatchSource:0}: Error finding container 10cf2a3ecaae009a3a198e0ec8d3b334a786503fd3d645dd404eb63baee7b479: Status 404 returned error can't find the container with id 10cf2a3ecaae009a3a198e0ec8d3b334a786503fd3d645dd404eb63baee7b479 Oct 30 02:15:01.071641 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:01.071609 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8f3eda_f86d_4b97_adb7_131d606bf442.slice/crio-6bb4415ab951e58f39d9b31fa01505d7b87f435d7882bff6f48e329e2498d906 WatchSource:0}: Error finding container 6bb4415ab951e58f39d9b31fa01505d7b87f435d7882bff6f48e329e2498d906: Status 404 returned error can't find the container with id 6bb4415ab951e58f39d9b31fa01505d7b87f435d7882bff6f48e329e2498d906 Oct 30 02:15:01.087317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.087278 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr"] Oct 30 02:15:01.094431 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:01.094395 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1884eb8f_8045_44a4_908c_13bbc9995d64.slice/crio-183d4f418988130d5731368aabb6f154a7d84edca9d78c9cd0329809727c64bc WatchSource:0}: Error finding container 183d4f418988130d5731368aabb6f154a7d84edca9d78c9cd0329809727c64bc: Status 404 returned error can't find the container with id 183d4f418988130d5731368aabb6f154a7d84edca9d78c9cd0329809727c64bc Oct 30 02:15:01.126157 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:01.126129 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429b5fb1_4e22_4e6d_ae49_0f0fd1972095.slice/crio-d231f0c69a8b1f2e042319c6e4110b9a7e27d3c726cb099ad19c3394d25f1064 WatchSource:0}: Error finding container d231f0c69a8b1f2e042319c6e4110b9a7e27d3c726cb099ad19c3394d25f1064: Status 404 returned error can't find the container with id d231f0c69a8b1f2e042319c6e4110b9a7e27d3c726cb099ad19c3394d25f1064 Oct 30 02:15:01.127085 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.127055 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb"] Oct 30 02:15:01.147788 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.147752 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm"] Oct 30 02:15:01.148848 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:01.148820 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99be1c56_597c_4776_b219_44b73eaebfae.slice/crio-0746bfceb9664bcb373ad38f6416822ad1efec18e1f600c2190ca62a44f3e4b2 WatchSource:0}: Error finding container 0746bfceb9664bcb373ad38f6416822ad1efec18e1f600c2190ca62a44f3e4b2: Status 404 returned error can't find the container with id 0746bfceb9664bcb373ad38f6416822ad1efec18e1f600c2190ca62a44f3e4b2 Oct 30 02:15:01.692492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.692454 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" event={"ID":"ad8f3eda-f86d-4b97-adb7-131d606bf442","Type":"ContainerStarted","Data":"eb1be9c42bdb56810ebf48f2673d7fce0f0dcade3de4a75253e51886c9bf8bb5"} Oct 30 02:15:01.692664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.692518 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" event={"ID":"ad8f3eda-f86d-4b97-adb7-131d606bf442","Type":"ContainerStarted","Data":"6bb4415ab951e58f39d9b31fa01505d7b87f435d7882bff6f48e329e2498d906"} Oct 30 02:15:01.694236 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.694204 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" event={"ID":"1884eb8f-8045-44a4-908c-13bbc9995d64","Type":"ContainerStarted","Data":"95fe8ffa907c714370e4922089c1f23271e07640fde41c86942b535c25c9a86a"} Oct 30 02:15:01.694368 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.694257 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" event={"ID":"1884eb8f-8045-44a4-908c-13bbc9995d64","Type":"ContainerStarted","Data":"183d4f418988130d5731368aabb6f154a7d84edca9d78c9cd0329809727c64bc"} Oct 30 02:15:01.696048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.696020 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" event={"ID":"99be1c56-597c-4776-b219-44b73eaebfae","Type":"ContainerStarted","Data":"497f30ba089d152dde27579a35d6d4afe25a57f572c20abc7e25cc5d51e96e2d"} Oct 30 02:15:01.696160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.696076 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" event={"ID":"99be1c56-597c-4776-b219-44b73eaebfae","Type":"ContainerStarted","Data":"0746bfceb9664bcb373ad38f6416822ad1efec18e1f600c2190ca62a44f3e4b2"} Oct 30 02:15:01.697742 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.697715 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" event={"ID":"459172b3-dfd3-46c0-9338-7640a59b6d44","Type":"ContainerStarted","Data":"b05fe0cff6a5a3372511993dd134effa2d396c54589ad4832642ea52a4c8d957"} Oct 30 02:15:01.697853 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.697758 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" event={"ID":"459172b3-dfd3-46c0-9338-7640a59b6d44","Type":"ContainerStarted","Data":"1e6c40a4e92dfe790609a2a9082d61387fa21a6bd7fb61a753f3be92f3efa60b"} Oct 30 02:15:01.699693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.699666 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" event={"ID":"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319","Type":"ContainerStarted","Data":"b19fd313d1730008029219279cf0c2eb4446507cc8d2203ddf9ccd0fa77e33c8"} Oct 30 02:15:01.699816 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.699714 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" event={"ID":"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319","Type":"ContainerStarted","Data":"10cf2a3ecaae009a3a198e0ec8d3b334a786503fd3d645dd404eb63baee7b479"} Oct 30 02:15:01.702039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.702017 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" event={"ID":"429b5fb1-4e22-4e6d-ae49-0f0fd1972095","Type":"ContainerStarted","Data":"2194fbd4f9db4bbcf287327523b155247823a70d32af2f38221a9397625d0cd2"} Oct 30 02:15:01.702162 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.702061 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" event={"ID":"429b5fb1-4e22-4e6d-ae49-0f0fd1972095","Type":"ContainerStarted","Data":"d231f0c69a8b1f2e042319c6e4110b9a7e27d3c726cb099ad19c3394d25f1064"} Oct 30 02:15:01.739830 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.739772 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" podStartSLOduration=1.7397570340000001 podStartE2EDuration="1.739757034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:01.737718532 +0000 UTC m=+2780.240650744" watchObservedRunningTime="2026-10-30 02:15:01.739757034 +0000 UTC m=+2780.242689248" Oct 30 02:15:01.784044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.783980 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" podStartSLOduration=1.7839640110000001 podStartE2EDuration="1.783964011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:01.781202768 +0000 UTC m=+2780.284134991" watchObservedRunningTime="2026-10-30 02:15:01.783964011 +0000 UTC m=+2780.286896256" Oct 30 02:15:01.827473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.827349 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" podStartSLOduration=1.82733209 podStartE2EDuration="1.82733209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:01.823853004 +0000 UTC m=+2780.326785226" watchObservedRunningTime="2026-10-30 02:15:01.82733209 +0000 UTC m=+2780.330264309" Oct 30 02:15:01.872469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.872409 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" podStartSLOduration=1.8723943250000001 podStartE2EDuration="1.872394325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:01.871425508 +0000 UTC m=+2780.374357728" watchObservedRunningTime="2026-10-30 02:15:01.872394325 +0000 UTC m=+2780.375326545" Oct 30 02:15:01.919512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.919453 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" podStartSLOduration=1.919435176 podStartE2EDuration="1.919435176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:01.915095257 +0000 UTC m=+2780.418027476" watchObservedRunningTime="2026-10-30 02:15:01.919435176 +0000 UTC m=+2780.422367420" Oct 30 02:15:01.962608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:01.962523 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" podStartSLOduration=1.962501108 podStartE2EDuration="1.962501108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:01.958724986 +0000 UTC m=+2780.461657208" watchObservedRunningTime="2026-10-30 02:15:01.962501108 +0000 UTC m=+2780.465433332" Oct 30 02:15:02.712859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.712823 5035 generic.go:356] "Generic (PLEG): container finished" podID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerID="95fe8ffa907c714370e4922089c1f23271e07640fde41c86942b535c25c9a86a" exitCode=60 Oct 30 02:15:02.713347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.712898 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" event={"ID":"1884eb8f-8045-44a4-908c-13bbc9995d64","Type":"ContainerDied","Data":"95fe8ffa907c714370e4922089c1f23271e07640fde41c86942b535c25c9a86a"} Oct 30 02:15:02.713347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.713245 5035 scope.go:122] "RemoveContainer" containerID="95fe8ffa907c714370e4922089c1f23271e07640fde41c86942b535c25c9a86a" Oct 30 02:15:02.714801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.714771 5035 generic.go:356] "Generic (PLEG): container finished" podID="99be1c56-597c-4776-b219-44b73eaebfae" containerID="497f30ba089d152dde27579a35d6d4afe25a57f572c20abc7e25cc5d51e96e2d" exitCode=60 Oct 30 02:15:02.714953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.714851 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" event={"ID":"99be1c56-597c-4776-b219-44b73eaebfae","Type":"ContainerDied","Data":"497f30ba089d152dde27579a35d6d4afe25a57f572c20abc7e25cc5d51e96e2d"} Oct 30 02:15:02.715123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.715101 5035 scope.go:122] "RemoveContainer" containerID="497f30ba089d152dde27579a35d6d4afe25a57f572c20abc7e25cc5d51e96e2d" Oct 30 02:15:02.716897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.716842 5035 generic.go:356] "Generic (PLEG): container finished" podID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerID="b05fe0cff6a5a3372511993dd134effa2d396c54589ad4832642ea52a4c8d957" exitCode=60 Oct 30 02:15:02.716975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.716916 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" event={"ID":"459172b3-dfd3-46c0-9338-7640a59b6d44","Type":"ContainerDied","Data":"b05fe0cff6a5a3372511993dd134effa2d396c54589ad4832642ea52a4c8d957"} Oct 30 02:15:02.717207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.717191 5035 scope.go:122] "RemoveContainer" containerID="b05fe0cff6a5a3372511993dd134effa2d396c54589ad4832642ea52a4c8d957" Oct 30 02:15:02.718967 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.718842 5035 generic.go:356] "Generic (PLEG): container finished" podID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerID="b19fd313d1730008029219279cf0c2eb4446507cc8d2203ddf9ccd0fa77e33c8" exitCode=60 Oct 30 02:15:02.718967 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.718914 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" event={"ID":"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319","Type":"ContainerDied","Data":"b19fd313d1730008029219279cf0c2eb4446507cc8d2203ddf9ccd0fa77e33c8"} Oct 30 02:15:02.719157 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.719139 5035 scope.go:122] "RemoveContainer" containerID="b19fd313d1730008029219279cf0c2eb4446507cc8d2203ddf9ccd0fa77e33c8" Oct 30 02:15:02.720790 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.720770 5035 generic.go:356] "Generic (PLEG): container finished" podID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerID="2194fbd4f9db4bbcf287327523b155247823a70d32af2f38221a9397625d0cd2" exitCode=60 Oct 30 02:15:02.720890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.720844 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" event={"ID":"429b5fb1-4e22-4e6d-ae49-0f0fd1972095","Type":"ContainerDied","Data":"2194fbd4f9db4bbcf287327523b155247823a70d32af2f38221a9397625d0cd2"} Oct 30 02:15:02.721099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.721080 5035 scope.go:122] "RemoveContainer" containerID="2194fbd4f9db4bbcf287327523b155247823a70d32af2f38221a9397625d0cd2" Oct 30 02:15:02.722459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.722440 5035 generic.go:356] "Generic (PLEG): container finished" podID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerID="eb1be9c42bdb56810ebf48f2673d7fce0f0dcade3de4a75253e51886c9bf8bb5" exitCode=60 Oct 30 02:15:02.722586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.722472 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" event={"ID":"ad8f3eda-f86d-4b97-adb7-131d606bf442","Type":"ContainerDied","Data":"eb1be9c42bdb56810ebf48f2673d7fce0f0dcade3de4a75253e51886c9bf8bb5"} Oct 30 02:15:02.722737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:02.722722 5035 scope.go:122] "RemoveContainer" containerID="eb1be9c42bdb56810ebf48f2673d7fce0f0dcade3de4a75253e51886c9bf8bb5" Oct 30 02:15:03.733868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:03.733826 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" event={"ID":"459172b3-dfd3-46c0-9338-7640a59b6d44","Type":"ContainerStarted","Data":"a031ea814e5a7a464cd0a40ae47faf274f76f39a27c7ce173913ead436fa87f2"} Oct 30 02:15:03.736012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:03.735982 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" event={"ID":"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319","Type":"ContainerStarted","Data":"0b80c9662278b0d91309571e94256a8db94249ac5cec5dbdc28c884a983d6296"} Oct 30 02:15:03.738128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:03.738102 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" event={"ID":"429b5fb1-4e22-4e6d-ae49-0f0fd1972095","Type":"ContainerStarted","Data":"03a9fd8ac24299e32b1f02ebc54cba13c177b98812eb1c10dc4bed023364eb16"} Oct 30 02:15:03.740150 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:03.740120 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" event={"ID":"ad8f3eda-f86d-4b97-adb7-131d606bf442","Type":"ContainerStarted","Data":"ae527f20dc8844584de930fc27f02aec1fbdae583a1e9c2523c53cb0d4089ae9"} Oct 30 02:15:03.742598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:03.742268 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" event={"ID":"1884eb8f-8045-44a4-908c-13bbc9995d64","Type":"ContainerStarted","Data":"8d559d04f29eb7d62afc9cf7f1108d031cb0f3d0bd64338095b5513a65f6f41a"} Oct 30 02:15:03.744420 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:03.744396 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" event={"ID":"99be1c56-597c-4776-b219-44b73eaebfae","Type":"ContainerStarted","Data":"f7ebf541efc75c2c8dd48a3b396ab0134d8db7313bc2877d7b23e8c27f8895a8"} Oct 30 02:15:04.355338 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.355294 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-46t2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:15:04.355567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.355372 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:15:04.755312 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.755276 5035 generic.go:356] "Generic (PLEG): container finished" podID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerID="8d559d04f29eb7d62afc9cf7f1108d031cb0f3d0bd64338095b5513a65f6f41a" exitCode=60 Oct 30 02:15:04.755931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.755359 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" event={"ID":"1884eb8f-8045-44a4-908c-13bbc9995d64","Type":"ContainerDied","Data":"8d559d04f29eb7d62afc9cf7f1108d031cb0f3d0bd64338095b5513a65f6f41a"} Oct 30 02:15:04.755931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.755436 5035 scope.go:122] "RemoveContainer" containerID="95fe8ffa907c714370e4922089c1f23271e07640fde41c86942b535c25c9a86a" Oct 30 02:15:04.755931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.755703 5035 scope.go:122] "RemoveContainer" containerID="8d559d04f29eb7d62afc9cf7f1108d031cb0f3d0bd64338095b5513a65f6f41a" Oct 30 02:15:04.756163 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:04.756131 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr_odh-ai-gateway-infra(1884eb8f-8045-44a4-908c-13bbc9995d64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" Oct 30 02:15:04.758000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.757978 5035 generic.go:356] "Generic (PLEG): container finished" podID="99be1c56-597c-4776-b219-44b73eaebfae" containerID="f7ebf541efc75c2c8dd48a3b396ab0134d8db7313bc2877d7b23e8c27f8895a8" exitCode=60 Oct 30 02:15:04.758105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.758045 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" event={"ID":"99be1c56-597c-4776-b219-44b73eaebfae","Type":"ContainerDied","Data":"f7ebf541efc75c2c8dd48a3b396ab0134d8db7313bc2877d7b23e8c27f8895a8"} Oct 30 02:15:04.758349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.758332 5035 scope.go:122] "RemoveContainer" containerID="f7ebf541efc75c2c8dd48a3b396ab0134d8db7313bc2877d7b23e8c27f8895a8" Oct 30 02:15:04.758718 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:04.758693 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm_odh-ai-gateway-infra(99be1c56-597c-4776-b219-44b73eaebfae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" podUID="99be1c56-597c-4776-b219-44b73eaebfae" Oct 30 02:15:04.760133 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.760107 5035 generic.go:356] "Generic (PLEG): container finished" podID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerID="a031ea814e5a7a464cd0a40ae47faf274f76f39a27c7ce173913ead436fa87f2" exitCode=60 Oct 30 02:15:04.760233 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.760184 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" event={"ID":"459172b3-dfd3-46c0-9338-7640a59b6d44","Type":"ContainerDied","Data":"a031ea814e5a7a464cd0a40ae47faf274f76f39a27c7ce173913ead436fa87f2"} Oct 30 02:15:04.760461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.760446 5035 scope.go:122] "RemoveContainer" containerID="a031ea814e5a7a464cd0a40ae47faf274f76f39a27c7ce173913ead436fa87f2" Oct 30 02:15:04.760839 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:04.760813 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw_odh-ai-gateway-infra(459172b3-dfd3-46c0-9338-7640a59b6d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" Oct 30 02:15:04.762510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.762489 5035 generic.go:356] "Generic (PLEG): container finished" podID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerID="0b80c9662278b0d91309571e94256a8db94249ac5cec5dbdc28c884a983d6296" exitCode=60 Oct 30 02:15:04.762632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.762584 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" event={"ID":"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319","Type":"ContainerDied","Data":"0b80c9662278b0d91309571e94256a8db94249ac5cec5dbdc28c884a983d6296"} Oct 30 02:15:04.762881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.762864 5035 scope.go:122] "RemoveContainer" containerID="0b80c9662278b0d91309571e94256a8db94249ac5cec5dbdc28c884a983d6296" Oct 30 02:15:04.763208 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:04.763189 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6_odh-ai-gateway-infra(4ea2f6d5-3cd2-4d6f-806e-de5e3f560319)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" Oct 30 02:15:04.764679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.764656 5035 generic.go:356] "Generic (PLEG): container finished" podID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerID="03a9fd8ac24299e32b1f02ebc54cba13c177b98812eb1c10dc4bed023364eb16" exitCode=60 Oct 30 02:15:04.764778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.764735 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" event={"ID":"429b5fb1-4e22-4e6d-ae49-0f0fd1972095","Type":"ContainerDied","Data":"03a9fd8ac24299e32b1f02ebc54cba13c177b98812eb1c10dc4bed023364eb16"} Oct 30 02:15:04.764992 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.764974 5035 scope.go:122] "RemoveContainer" containerID="03a9fd8ac24299e32b1f02ebc54cba13c177b98812eb1c10dc4bed023364eb16" Oct 30 02:15:04.765320 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:04.765296 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb_odh-ai-gateway-infra(429b5fb1-4e22-4e6d-ae49-0f0fd1972095)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" Oct 30 02:15:04.766939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.766919 5035 generic.go:356] "Generic (PLEG): container finished" podID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerID="ae527f20dc8844584de930fc27f02aec1fbdae583a1e9c2523c53cb0d4089ae9" exitCode=60 Oct 30 02:15:04.767035 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.766992 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" event={"ID":"ad8f3eda-f86d-4b97-adb7-131d606bf442","Type":"ContainerDied","Data":"ae527f20dc8844584de930fc27f02aec1fbdae583a1e9c2523c53cb0d4089ae9"} Oct 30 02:15:04.767237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.767223 5035 scope.go:122] "RemoveContainer" containerID="ae527f20dc8844584de930fc27f02aec1fbdae583a1e9c2523c53cb0d4089ae9" Oct 30 02:15:04.767649 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:04.767621 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4_odh-ai-gateway-infra(ad8f3eda-f86d-4b97-adb7-131d606bf442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" Oct 30 02:15:04.771263 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.771243 5035 scope.go:122] "RemoveContainer" containerID="497f30ba089d152dde27579a35d6d4afe25a57f572c20abc7e25cc5d51e96e2d" Oct 30 02:15:04.786224 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.786198 5035 scope.go:122] "RemoveContainer" containerID="b05fe0cff6a5a3372511993dd134effa2d396c54589ad4832642ea52a4c8d957" Oct 30 02:15:04.801456 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.801429 5035 scope.go:122] "RemoveContainer" containerID="b19fd313d1730008029219279cf0c2eb4446507cc8d2203ddf9ccd0fa77e33c8" Oct 30 02:15:04.826588 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.826555 5035 scope.go:122] "RemoveContainer" containerID="2194fbd4f9db4bbcf287327523b155247823a70d32af2f38221a9397625d0cd2" Oct 30 02:15:04.843083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:04.843056 5035 scope.go:122] "RemoveContainer" containerID="eb1be9c42bdb56810ebf48f2673d7fce0f0dcade3de4a75253e51886c9bf8bb5" Oct 30 02:15:05.202317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.202283 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 02:15:05.339478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339387 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-envoy\") pod \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " Oct 30 02:15:05.339478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339438 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-credential-socket\") pod \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " Oct 30 02:15:05.339697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339511 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-certs\") pod \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " Oct 30 02:15:05.339697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339599 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-socket\") pod \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " Oct 30 02:15:05.339697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339629 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-data\") pod \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " Oct 30 02:15:05.339856 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339718 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-podinfo\") pod \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " Oct 30 02:15:05.339941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339819 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-credential-socket" pod "b49859e7-272b-41fb-9950-c85bf8cdb9ea" (UID: "b49859e7-272b-41fb-9950-c85bf8cdb9ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:05.339941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339841 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istiod-ca-cert\") pod \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " Oct 30 02:15:05.340056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339930 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-socket" pod "b49859e7-272b-41fb-9950-c85bf8cdb9ea" (UID: "b49859e7-272b-41fb-9950-c85bf8cdb9ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:05.340056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339937 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-certs" pod "b49859e7-272b-41fb-9950-c85bf8cdb9ea" (UID: "b49859e7-272b-41fb-9950-c85bf8cdb9ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:05.340056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.339982 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-data" pod "b49859e7-272b-41fb-9950-c85bf8cdb9ea" (UID: "b49859e7-272b-41fb-9950-c85bf8cdb9ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:05.340056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.340015 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-token\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-token\") pod \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " Oct 30 02:15:05.340246 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.340101 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-kube-api-access-x2qq2\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-kube-api-access-x2qq2\") pod \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\" (UID: \"b49859e7-272b-41fb-9950-c85bf8cdb9ea\") " Oct 30 02:15:05.340246 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.340203 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istiod-ca-cert" pod "b49859e7-272b-41fb-9950-c85bf8cdb9ea" (UID: "b49859e7-272b-41fb-9950-c85bf8cdb9ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:05.340838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.340809 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:05.340838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.340840 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:05.341071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.340854 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:05.341071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.340868 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:05.341071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.340882 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:05.341999 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.341968 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-envoy" pod "b49859e7-272b-41fb-9950-c85bf8cdb9ea" (UID: "b49859e7-272b-41fb-9950-c85bf8cdb9ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:05.342117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.342039 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-token" pod "b49859e7-272b-41fb-9950-c85bf8cdb9ea" (UID: "b49859e7-272b-41fb-9950-c85bf8cdb9ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:05.342117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.342069 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-podinfo" pod "b49859e7-272b-41fb-9950-c85bf8cdb9ea" (UID: "b49859e7-272b-41fb-9950-c85bf8cdb9ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:15:05.342317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.342292 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-kube-api-access-x2qq2" pod "b49859e7-272b-41fb-9950-c85bf8cdb9ea" (UID: "b49859e7-272b-41fb-9950-c85bf8cdb9ea"). InnerVolumeSpecName "kube-api-access-x2qq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:05.442381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.442343 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:05.442381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.442376 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:05.442381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.442387 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2qq2\" (UniqueName: \"kubernetes.io/projected/b49859e7-272b-41fb-9950-c85bf8cdb9ea-kube-api-access-x2qq2\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:05.442705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.442397 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49859e7-272b-41fb-9950-c85bf8cdb9ea-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:05.777364 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.777302 5035 scope.go:122] "RemoveContainer" containerID="a031ea814e5a7a464cd0a40ae47faf274f76f39a27c7ce173913ead436fa87f2" Oct 30 02:15:05.777857 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:05.777829 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw_odh-ai-gateway-infra(459172b3-dfd3-46c0-9338-7640a59b6d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" Oct 30 02:15:05.779082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.779057 5035 scope.go:122] "RemoveContainer" containerID="0b80c9662278b0d91309571e94256a8db94249ac5cec5dbdc28c884a983d6296" Oct 30 02:15:05.779443 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:05.779408 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6_odh-ai-gateway-infra(4ea2f6d5-3cd2-4d6f-806e-de5e3f560319)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" Oct 30 02:15:05.780710 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.780693 5035 scope.go:122] "RemoveContainer" containerID="03a9fd8ac24299e32b1f02ebc54cba13c177b98812eb1c10dc4bed023364eb16" Oct 30 02:15:05.781036 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:05.781015 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb_odh-ai-gateway-infra(429b5fb1-4e22-4e6d-ae49-0f0fd1972095)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" Oct 30 02:15:05.782433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.782415 5035 scope.go:122] "RemoveContainer" containerID="ae527f20dc8844584de930fc27f02aec1fbdae583a1e9c2523c53cb0d4089ae9" Oct 30 02:15:05.782764 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:05.782743 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4_odh-ai-gateway-infra(ad8f3eda-f86d-4b97-adb7-131d606bf442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" Oct 30 02:15:05.784156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.784134 5035 scope.go:122] "RemoveContainer" containerID="8d559d04f29eb7d62afc9cf7f1108d031cb0f3d0bd64338095b5513a65f6f41a" Oct 30 02:15:05.784437 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:05.784417 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr_odh-ai-gateway-infra(1884eb8f-8045-44a4-908c-13bbc9995d64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" Oct 30 02:15:05.785225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.785200 5035 generic.go:356] "Generic (PLEG): container finished" podID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerID="e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a" exitCode=0 Oct 30 02:15:05.785294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.785237 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" Oct 30 02:15:05.785294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.785239 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" event={"ID":"b49859e7-272b-41fb-9950-c85bf8cdb9ea","Type":"ContainerDied","Data":"e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a"} Oct 30 02:15:05.785397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.785296 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r" event={"ID":"b49859e7-272b-41fb-9950-c85bf8cdb9ea","Type":"ContainerDied","Data":"47dbf8658948ca2bb011e3fcbf9621ef4d89b30b6e2702443e722a09b377573a"} Oct 30 02:15:05.785397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.785326 5035 scope.go:122] "RemoveContainer" containerID="e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a" Oct 30 02:15:05.788142 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.788111 5035 scope.go:122] "RemoveContainer" containerID="f7ebf541efc75c2c8dd48a3b396ab0134d8db7313bc2877d7b23e8c27f8895a8" Oct 30 02:15:05.788395 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:05.788372 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm_odh-ai-gateway-infra(99be1c56-597c-4776-b219-44b73eaebfae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" podUID="99be1c56-597c-4776-b219-44b73eaebfae" Oct 30 02:15:05.797593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.797571 5035 scope.go:122] "RemoveContainer" containerID="e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a" Oct 30 02:15:05.797943 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:05.797919 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a\": container with ID starting with e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a not found: ID does not exist" containerID="e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a" Oct 30 02:15:05.798016 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:05.797953 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a"} err="failed to get container status \"e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a\": rpc error: code = NotFound desc = could not find container \"e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a\": container with ID starting with e5303c476b1073f93f7dd3bd73dcabe4edbe8028b34198c117574d44631ce68a not found: ID does not exist" Oct 30 02:15:06.115206 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:06.115155 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r"] Oct 30 02:15:06.122725 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:06.122686 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-46t2r"] Oct 30 02:15:07.646788 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.646753 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" path="/var/lib/kubelet/pods/b49859e7-272b-41fb-9950-c85bf8cdb9ea/volumes" Oct 30 02:15:07.648659 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.648631 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5"] Oct 30 02:15:07.649409 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.649393 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" Oct 30 02:15:07.649459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.649410 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" Oct 30 02:15:07.649559 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.649548 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49859e7-272b-41fb-9950-c85bf8cdb9ea" containerName="istio-proxy" Oct 30 02:15:07.654905 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.654881 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.676807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.676773 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0fdc4e-serving-cert\"" Oct 30 02:15:07.707624 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.707337 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5"] Oct 30 02:15:07.773064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.773010 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpd9n\" (UniqueName: \"kubernetes.io/projected/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-kube-api-access-lpd9n\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.773257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.773221 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.773306 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.773286 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-metrics-certs\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.773376 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.773355 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.875048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.875006 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.875048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.875062 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-metrics-certs\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.875315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.875102 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.875315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.875222 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpd9n\" (UniqueName: \"kubernetes.io/projected/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-kube-api-access-lpd9n\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.875880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.875849 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.877997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.877969 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-metrics-certs\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.878106 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.878046 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.902781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.902704 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpd9n\" (UniqueName: \"kubernetes.io/projected/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-kube-api-access-lpd9n\") pod \"maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:07.971910 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:07.971826 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:08.398319 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:08.398282 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5"] Oct 30 02:15:08.399773 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:08.399741 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19dfbcf_ada7_4a8e_b41b_9432ebe58e3d.slice/crio-f8aefff9ae8d5fe46c4d7315a4c0836a4cf1c1dec08d95d0d0050ac84c75eebb WatchSource:0}: Error finding container f8aefff9ae8d5fe46c4d7315a4c0836a4cf1c1dec08d95d0d0050ac84c75eebb: Status 404 returned error can't find the container with id f8aefff9ae8d5fe46c4d7315a4c0836a4cf1c1dec08d95d0d0050ac84c75eebb Oct 30 02:15:08.819112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:08.819082 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" event={"ID":"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d","Type":"ContainerStarted","Data":"f8aefff9ae8d5fe46c4d7315a4c0836a4cf1c1dec08d95d0d0050ac84c75eebb"} Oct 30 02:15:09.828918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:09.828880 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" event={"ID":"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d","Type":"ContainerStarted","Data":"4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d"} Oct 30 02:15:09.829405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:09.829100 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:09.885494 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:09.885432 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" podStartSLOduration=2.553433504 podStartE2EDuration="2.885415525s" totalImagesPullingTime="331.982021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:09.877207722 +0000 UTC m=+2788.380139982" watchObservedRunningTime="2026-10-30 02:15:09.885415525 +0000 UTC m=+2788.388347745" Oct 30 02:15:11.007605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.007562 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj"] Oct 30 02:15:11.012498 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.012471 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:11.036413 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.036372 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj"] Oct 30 02:15:11.043419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.043376 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-9dfb3c-dockercfg-9lv46\"" Oct 30 02:15:11.043419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.043377 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-9dfb3c\"" Oct 30 02:15:11.129689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.129642 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802465ef-c6e6-40eb-b083-c3fa61aed348-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj\" (UID: \"802465ef-c6e6-40eb-b083-c3fa61aed348\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:11.133993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.133943 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgj6q\" (UniqueName: \"kubernetes.io/projected/802465ef-c6e6-40eb-b083-c3fa61aed348-kube-api-access-bgj6q\") pod \"payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj\" (UID: \"802465ef-c6e6-40eb-b083-c3fa61aed348\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:11.235521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.235478 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802465ef-c6e6-40eb-b083-c3fa61aed348-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj\" (UID: \"802465ef-c6e6-40eb-b083-c3fa61aed348\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:11.235769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.235751 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgj6q\" (UniqueName: \"kubernetes.io/projected/802465ef-c6e6-40eb-b083-c3fa61aed348-kube-api-access-bgj6q\") pod \"payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj\" (UID: \"802465ef-c6e6-40eb-b083-c3fa61aed348\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:11.236285 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.236265 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802465ef-c6e6-40eb-b083-c3fa61aed348-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj\" (UID: \"802465ef-c6e6-40eb-b083-c3fa61aed348\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:11.261685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.261605 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgj6q\" (UniqueName: \"kubernetes.io/projected/802465ef-c6e6-40eb-b083-c3fa61aed348-kube-api-access-bgj6q\") pod \"payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj\" (UID: \"802465ef-c6e6-40eb-b083-c3fa61aed348\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:11.335786 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.335744 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:11.649623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.649587 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj"] Oct 30 02:15:11.650466 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:11.650434 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802465ef_c6e6_40eb_b083_c3fa61aed348.slice/crio-a636a075ba7e5d7f725452ae533195ab9e8734333c0e7dec93da9c9320beb4c5 WatchSource:0}: Error finding container a636a075ba7e5d7f725452ae533195ab9e8734333c0e7dec93da9c9320beb4c5: Status 404 returned error can't find the container with id a636a075ba7e5d7f725452ae533195ab9e8734333c0e7dec93da9c9320beb4c5 Oct 30 02:15:11.847329 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.847291 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" event={"ID":"802465ef-c6e6-40eb-b083-c3fa61aed348","Type":"ContainerStarted","Data":"b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1"} Oct 30 02:15:11.847586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.847384 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" event={"ID":"802465ef-c6e6-40eb-b083-c3fa61aed348","Type":"ContainerStarted","Data":"a636a075ba7e5d7f725452ae533195ab9e8734333c0e7dec93da9c9320beb4c5"} Oct 30 02:15:11.847586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.847551 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:11.897912 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:11.897785 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" podStartSLOduration=1.8977686839999999 podStartE2EDuration="1.897768684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:11.893190062 +0000 UTC m=+2790.396122282" watchObservedRunningTime="2026-10-30 02:15:11.897768684 +0000 UTC m=+2790.400700903" Oct 30 02:15:15.635396 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:15.635357 5035 scope.go:122] "RemoveContainer" containerID="ae527f20dc8844584de930fc27f02aec1fbdae583a1e9c2523c53cb0d4089ae9" Oct 30 02:15:15.635863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:15.635458 5035 scope.go:122] "RemoveContainer" containerID="8d559d04f29eb7d62afc9cf7f1108d031cb0f3d0bd64338095b5513a65f6f41a" Oct 30 02:15:15.843483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:15.843449 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:15:16.635817 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.635777 5035 scope.go:122] "RemoveContainer" containerID="03a9fd8ac24299e32b1f02ebc54cba13c177b98812eb1c10dc4bed023364eb16" Oct 30 02:15:16.636315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.636094 5035 scope.go:122] "RemoveContainer" containerID="0b80c9662278b0d91309571e94256a8db94249ac5cec5dbdc28c884a983d6296" Oct 30 02:15:16.636315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.636183 5035 scope.go:122] "RemoveContainer" containerID="a031ea814e5a7a464cd0a40ae47faf274f76f39a27c7ce173913ead436fa87f2" Oct 30 02:15:16.636315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.636282 5035 scope.go:122] "RemoveContainer" containerID="f7ebf541efc75c2c8dd48a3b396ab0134d8db7313bc2877d7b23e8c27f8895a8" Oct 30 02:15:16.896139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.896100 5035 generic.go:356] "Generic (PLEG): container finished" podID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerID="fd65bee7880df242ff10bebeddc78656eae9eab915e507f7045a8c4b1021b386" exitCode=60 Oct 30 02:15:16.896319 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.896174 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" event={"ID":"ad8f3eda-f86d-4b97-adb7-131d606bf442","Type":"ContainerDied","Data":"fd65bee7880df242ff10bebeddc78656eae9eab915e507f7045a8c4b1021b386"} Oct 30 02:15:16.896319 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.896249 5035 scope.go:122] "RemoveContainer" containerID="ae527f20dc8844584de930fc27f02aec1fbdae583a1e9c2523c53cb0d4089ae9" Oct 30 02:15:16.896844 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.896812 5035 scope.go:122] "RemoveContainer" containerID="fd65bee7880df242ff10bebeddc78656eae9eab915e507f7045a8c4b1021b386" Oct 30 02:15:16.897266 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:16.897239 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4_odh-ai-gateway-infra(ad8f3eda-f86d-4b97-adb7-131d606bf442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" Oct 30 02:15:16.899278 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.899252 5035 generic.go:356] "Generic (PLEG): container finished" podID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerID="71204858bdbc9e115ce7e36e56a28b15c735b137a8131fc8bdcb3a1fb79510e2" exitCode=60 Oct 30 02:15:16.899398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.899336 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" event={"ID":"1884eb8f-8045-44a4-908c-13bbc9995d64","Type":"ContainerDied","Data":"71204858bdbc9e115ce7e36e56a28b15c735b137a8131fc8bdcb3a1fb79510e2"} Oct 30 02:15:16.899879 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:16.899859 5035 scope.go:122] "RemoveContainer" containerID="71204858bdbc9e115ce7e36e56a28b15c735b137a8131fc8bdcb3a1fb79510e2" Oct 30 02:15:16.900228 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:16.900209 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr_odh-ai-gateway-infra(1884eb8f-8045-44a4-908c-13bbc9995d64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" Oct 30 02:15:17.031112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.030826 5035 scope.go:122] "RemoveContainer" containerID="8d559d04f29eb7d62afc9cf7f1108d031cb0f3d0bd64338095b5513a65f6f41a" Oct 30 02:15:17.638751 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:17.638701 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:15:17.910390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.910295 5035 generic.go:356] "Generic (PLEG): container finished" podID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerID="209d85d6411c97b06ed1a8227ff3e00610fc00f1e5b1def826c7bd8a85115677" exitCode=60 Oct 30 02:15:17.910390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.910360 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" event={"ID":"459172b3-dfd3-46c0-9338-7640a59b6d44","Type":"ContainerDied","Data":"209d85d6411c97b06ed1a8227ff3e00610fc00f1e5b1def826c7bd8a85115677"} Oct 30 02:15:17.910674 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.910425 5035 scope.go:122] "RemoveContainer" containerID="a031ea814e5a7a464cd0a40ae47faf274f76f39a27c7ce173913ead436fa87f2" Oct 30 02:15:17.910951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.910931 5035 scope.go:122] "RemoveContainer" containerID="209d85d6411c97b06ed1a8227ff3e00610fc00f1e5b1def826c7bd8a85115677" Oct 30 02:15:17.911381 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:17.911354 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw_odh-ai-gateway-infra(459172b3-dfd3-46c0-9338-7640a59b6d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" Oct 30 02:15:17.913028 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.913000 5035 generic.go:356] "Generic (PLEG): container finished" podID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerID="5f71c5a66cfe7eb58aa2d3f6184ab316244a0c5f4bec24583dab2dd50adcea51" exitCode=60 Oct 30 02:15:17.913333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.913038 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" event={"ID":"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319","Type":"ContainerDied","Data":"5f71c5a66cfe7eb58aa2d3f6184ab316244a0c5f4bec24583dab2dd50adcea51"} Oct 30 02:15:17.913493 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.913477 5035 scope.go:122] "RemoveContainer" containerID="5f71c5a66cfe7eb58aa2d3f6184ab316244a0c5f4bec24583dab2dd50adcea51" Oct 30 02:15:17.913898 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:17.913860 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6_odh-ai-gateway-infra(4ea2f6d5-3cd2-4d6f-806e-de5e3f560319)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" Oct 30 02:15:17.916062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.916039 5035 generic.go:356] "Generic (PLEG): container finished" podID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerID="d3d5fd47e217cca50bd5e18e4b7d34243042775386b14673f0f12f9871f16518" exitCode=60 Oct 30 02:15:17.916180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.916100 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" event={"ID":"429b5fb1-4e22-4e6d-ae49-0f0fd1972095","Type":"ContainerDied","Data":"d3d5fd47e217cca50bd5e18e4b7d34243042775386b14673f0f12f9871f16518"} Oct 30 02:15:17.916600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.916577 5035 scope.go:122] "RemoveContainer" containerID="d3d5fd47e217cca50bd5e18e4b7d34243042775386b14673f0f12f9871f16518" Oct 30 02:15:17.916972 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:17.916936 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb_odh-ai-gateway-infra(429b5fb1-4e22-4e6d-ae49-0f0fd1972095)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" Oct 30 02:15:17.921978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.921953 5035 generic.go:356] "Generic (PLEG): container finished" podID="99be1c56-597c-4776-b219-44b73eaebfae" containerID="e269252d340bcf2b12f8c346ef0af07b491ce3d6f17a6bbbac9823c810c2f990" exitCode=60 Oct 30 02:15:17.922079 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.922011 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" event={"ID":"99be1c56-597c-4776-b219-44b73eaebfae","Type":"ContainerDied","Data":"e269252d340bcf2b12f8c346ef0af07b491ce3d6f17a6bbbac9823c810c2f990"} Oct 30 02:15:17.922389 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.922376 5035 scope.go:122] "RemoveContainer" containerID="e269252d340bcf2b12f8c346ef0af07b491ce3d6f17a6bbbac9823c810c2f990" Oct 30 02:15:17.922731 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:17.922715 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm_odh-ai-gateway-infra(99be1c56-597c-4776-b219-44b73eaebfae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" podUID="99be1c56-597c-4776-b219-44b73eaebfae" Oct 30 02:15:17.955397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:17.955358 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4"] Oct 30 02:15:18.030146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.030106 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr"] Oct 30 02:15:18.036325 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.036298 5035 scope.go:122] "RemoveContainer" containerID="0b80c9662278b0d91309571e94256a8db94249ac5cec5dbdc28c884a983d6296" Oct 30 02:15:18.068342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.068316 5035 scope.go:122] "RemoveContainer" containerID="03a9fd8ac24299e32b1f02ebc54cba13c177b98812eb1c10dc4bed023364eb16" Oct 30 02:15:18.092089 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.091906 5035 scope.go:122] "RemoveContainer" containerID="f7ebf541efc75c2c8dd48a3b396ab0134d8db7313bc2877d7b23e8c27f8895a8" Oct 30 02:15:18.183564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.183453 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:18.209113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.209084 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:18.302292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.302109 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1884eb8f-8045-44a4-908c-13bbc9995d64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1884eb8f-8045-44a4-908c-13bbc9995d64-maas-api-service-ca\") pod \"1884eb8f-8045-44a4-908c-13bbc9995d64\" (UID: \"1884eb8f-8045-44a4-908c-13bbc9995d64\") " Oct 30 02:15:18.302292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.302168 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1884eb8f-8045-44a4-908c-13bbc9995d64-kube-api-access-67hjk\" (UniqueName: \"kubernetes.io/projected/1884eb8f-8045-44a4-908c-13bbc9995d64-kube-api-access-67hjk\") pod \"1884eb8f-8045-44a4-908c-13bbc9995d64\" (UID: \"1884eb8f-8045-44a4-908c-13bbc9995d64\") " Oct 30 02:15:18.302292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.302273 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8f3eda-f86d-4b97-adb7-131d606bf442-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8f3eda-f86d-4b97-adb7-131d606bf442-maas-api-service-ca\") pod \"ad8f3eda-f86d-4b97-adb7-131d606bf442\" (UID: \"ad8f3eda-f86d-4b97-adb7-131d606bf442\") " Oct 30 02:15:18.302934 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.302424 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8f3eda-f86d-4b97-adb7-131d606bf442-kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/ad8f3eda-f86d-4b97-adb7-131d606bf442-kube-api-access-7slnv\") pod \"ad8f3eda-f86d-4b97-adb7-131d606bf442\" (UID: \"ad8f3eda-f86d-4b97-adb7-131d606bf442\") " Oct 30 02:15:18.303851 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.303728 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8f3eda-f86d-4b97-adb7-131d606bf442-maas-api-service-ca" pod "ad8f3eda-f86d-4b97-adb7-131d606bf442" (UID: "ad8f3eda-f86d-4b97-adb7-131d606bf442"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:18.303851 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.303810 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1884eb8f-8045-44a4-908c-13bbc9995d64-maas-api-service-ca" pod "1884eb8f-8045-44a4-908c-13bbc9995d64" (UID: "1884eb8f-8045-44a4-908c-13bbc9995d64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:18.310647 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.307724 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8f3eda-f86d-4b97-adb7-131d606bf442-kube-api-access-7slnv" pod "ad8f3eda-f86d-4b97-adb7-131d606bf442" (UID: "ad8f3eda-f86d-4b97-adb7-131d606bf442"). InnerVolumeSpecName "kube-api-access-7slnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:18.310647 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.309928 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1884eb8f-8045-44a4-908c-13bbc9995d64-kube-api-access-67hjk" pod "1884eb8f-8045-44a4-908c-13bbc9995d64" (UID: "1884eb8f-8045-44a4-908c-13bbc9995d64"). InnerVolumeSpecName "kube-api-access-67hjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:18.404944 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.404901 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8f3eda-f86d-4b97-adb7-131d606bf442-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:18.404944 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.404939 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/ad8f3eda-f86d-4b97-adb7-131d606bf442-kube-api-access-7slnv\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:18.404944 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.404952 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1884eb8f-8045-44a4-908c-13bbc9995d64-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:18.405267 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.404961 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67hjk\" (UniqueName: \"kubernetes.io/projected/1884eb8f-8045-44a4-908c-13bbc9995d64-kube-api-access-67hjk\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:18.936603 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.936559 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" event={"ID":"ad8f3eda-f86d-4b97-adb7-131d606bf442","Type":"ContainerDied","Data":"6bb4415ab951e58f39d9b31fa01505d7b87f435d7882bff6f48e329e2498d906"} Oct 30 02:15:18.937095 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.936572 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4" Oct 30 02:15:18.937095 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.936634 5035 scope.go:122] "RemoveContainer" containerID="fd65bee7880df242ff10bebeddc78656eae9eab915e507f7045a8c4b1021b386" Oct 30 02:15:18.938262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.938222 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" Oct 30 02:15:18.938262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.938234 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr" event={"ID":"1884eb8f-8045-44a4-908c-13bbc9995d64","Type":"ContainerDied","Data":"183d4f418988130d5731368aabb6f154a7d84edca9d78c9cd0329809727c64bc"} Oct 30 02:15:18.949830 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.949233 5035 scope.go:122] "RemoveContainer" containerID="71204858bdbc9e115ce7e36e56a28b15c735b137a8131fc8bdcb3a1fb79510e2" Oct 30 02:15:18.972002 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.971966 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw"] Oct 30 02:15:18.997254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:18.997215 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4"] Oct 30 02:15:19.005834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.005796 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888775-pqjt4"] Oct 30 02:15:19.034837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.034751 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm"] Oct 30 02:15:19.054388 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.053997 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr"] Oct 30 02:15:19.078345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.078273 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888775-7lvxr"] Oct 30 02:15:19.083387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.083352 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb"] Oct 30 02:15:19.136365 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.136289 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6"] Oct 30 02:15:19.185795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.185699 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:19.320238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.320197 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459172b3-dfd3-46c0-9338-7640a59b6d44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/459172b3-dfd3-46c0-9338-7640a59b6d44-maas-api-service-ca\") pod \"459172b3-dfd3-46c0-9338-7640a59b6d44\" (UID: \"459172b3-dfd3-46c0-9338-7640a59b6d44\") " Oct 30 02:15:19.320705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.320672 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459172b3-dfd3-46c0-9338-7640a59b6d44-maas-api-service-ca" pod "459172b3-dfd3-46c0-9338-7640a59b6d44" (UID: "459172b3-dfd3-46c0-9338-7640a59b6d44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.321203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.321170 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459172b3-dfd3-46c0-9338-7640a59b6d44-kube-api-access-pfgqs\" (UniqueName: \"kubernetes.io/projected/459172b3-dfd3-46c0-9338-7640a59b6d44-kube-api-access-pfgqs\") pod \"459172b3-dfd3-46c0-9338-7640a59b6d44\" (UID: \"459172b3-dfd3-46c0-9338-7640a59b6d44\") " Oct 30 02:15:19.322002 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.321966 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/459172b3-dfd3-46c0-9338-7640a59b6d44-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.324434 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.324404 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459172b3-dfd3-46c0-9338-7640a59b6d44-kube-api-access-pfgqs" pod "459172b3-dfd3-46c0-9338-7640a59b6d44" (UID: "459172b3-dfd3-46c0-9338-7640a59b6d44"). InnerVolumeSpecName "kube-api-access-pfgqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.329206 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.329179 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:19.338309 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.338281 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:19.343705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.343680 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:19.422486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.422453 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-kube-api-access-nmbhn\" (UniqueName: \"kubernetes.io/projected/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-kube-api-access-nmbhn\") pod \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\" (UID: \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\") " Oct 30 02:15:19.422686 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.422568 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99be1c56-597c-4776-b219-44b73eaebfae-kube-api-access-jx2l8\" (UniqueName: \"kubernetes.io/projected/99be1c56-597c-4776-b219-44b73eaebfae-kube-api-access-jx2l8\") pod \"99be1c56-597c-4776-b219-44b73eaebfae\" (UID: \"99be1c56-597c-4776-b219-44b73eaebfae\") " Oct 30 02:15:19.422766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.422699 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-maas-api-service-ca\") pod \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\" (UID: \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\") " Oct 30 02:15:19.422766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.422741 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-maas-api-service-ca\") pod \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\" (UID: \"429b5fb1-4e22-4e6d-ae49-0f0fd1972095\") " Oct 30 02:15:19.422861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.422790 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-kube-api-access-gplsk\" (UniqueName: \"kubernetes.io/projected/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-kube-api-access-gplsk\") pod \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\" (UID: \"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319\") " Oct 30 02:15:19.422913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.422876 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99be1c56-597c-4776-b219-44b73eaebfae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be1c56-597c-4776-b219-44b73eaebfae-maas-api-service-ca\") pod \"99be1c56-597c-4776-b219-44b73eaebfae\" (UID: \"99be1c56-597c-4776-b219-44b73eaebfae\") " Oct 30 02:15:19.423268 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.423241 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-maas-api-service-ca" pod "429b5fb1-4e22-4e6d-ae49-0f0fd1972095" (UID: "429b5fb1-4e22-4e6d-ae49-0f0fd1972095"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.427405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.423597 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99be1c56-597c-4776-b219-44b73eaebfae-maas-api-service-ca" pod "99be1c56-597c-4776-b219-44b73eaebfae" (UID: "99be1c56-597c-4776-b219-44b73eaebfae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.427405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.423240 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-maas-api-service-ca" pod "4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" (UID: "4ea2f6d5-3cd2-4d6f-806e-de5e3f560319"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.427405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.424075 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfgqs\" (UniqueName: \"kubernetes.io/projected/459172b3-dfd3-46c0-9338-7640a59b6d44-kube-api-access-pfgqs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.427405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.424093 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.427405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.424101 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.427405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.424110 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99be1c56-597c-4776-b219-44b73eaebfae-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.430056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.430022 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-kube-api-access-nmbhn" pod "429b5fb1-4e22-4e6d-ae49-0f0fd1972095" (UID: "429b5fb1-4e22-4e6d-ae49-0f0fd1972095"). InnerVolumeSpecName "kube-api-access-nmbhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.430752 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.430724 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99be1c56-597c-4776-b219-44b73eaebfae-kube-api-access-jx2l8" pod "99be1c56-597c-4776-b219-44b73eaebfae" (UID: "99be1c56-597c-4776-b219-44b73eaebfae"). InnerVolumeSpecName "kube-api-access-jx2l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.432485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.432168 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-kube-api-access-gplsk" pod "4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" (UID: "4ea2f6d5-3cd2-4d6f-806e-de5e3f560319"). InnerVolumeSpecName "kube-api-access-gplsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.525130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.525086 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gplsk\" (UniqueName: \"kubernetes.io/projected/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319-kube-api-access-gplsk\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.525130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.525122 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbhn\" (UniqueName: \"kubernetes.io/projected/429b5fb1-4e22-4e6d-ae49-0f0fd1972095-kube-api-access-nmbhn\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.525130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.525132 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx2l8\" (UniqueName: \"kubernetes.io/projected/99be1c56-597c-4776-b219-44b73eaebfae-kube-api-access-jx2l8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.645081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.644995 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" path="/var/lib/kubelet/pods/1884eb8f-8045-44a4-908c-13bbc9995d64/volumes" Oct 30 02:15:19.645511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.645497 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" path="/var/lib/kubelet/pods/ad8f3eda-f86d-4b97-adb7-131d606bf442/volumes" Oct 30 02:15:19.950226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.950194 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" Oct 30 02:15:19.950695 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.950190 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw" event={"ID":"459172b3-dfd3-46c0-9338-7640a59b6d44","Type":"ContainerDied","Data":"1e6c40a4e92dfe790609a2a9082d61387fa21a6bd7fb61a753f3be92f3efa60b"} Oct 30 02:15:19.950695 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.950326 5035 scope.go:122] "RemoveContainer" containerID="209d85d6411c97b06ed1a8227ff3e00610fc00f1e5b1def826c7bd8a85115677" Oct 30 02:15:19.952450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.952124 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" event={"ID":"4ea2f6d5-3cd2-4d6f-806e-de5e3f560319","Type":"ContainerDied","Data":"10cf2a3ecaae009a3a198e0ec8d3b334a786503fd3d645dd404eb63baee7b479"} Oct 30 02:15:19.952450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.952229 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6" Oct 30 02:15:19.954489 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.954466 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" Oct 30 02:15:19.954658 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.954478 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb" event={"ID":"429b5fb1-4e22-4e6d-ae49-0f0fd1972095","Type":"ContainerDied","Data":"d231f0c69a8b1f2e042319c6e4110b9a7e27d3c726cb099ad19c3394d25f1064"} Oct 30 02:15:19.959548 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.959497 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" event={"ID":"99be1c56-597c-4776-b219-44b73eaebfae","Type":"ContainerDied","Data":"0746bfceb9664bcb373ad38f6416822ad1efec18e1f600c2190ca62a44f3e4b2"} Oct 30 02:15:19.959673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.959615 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm" Oct 30 02:15:19.965327 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.965299 5035 scope.go:122] "RemoveContainer" containerID="5f71c5a66cfe7eb58aa2d3f6184ab316244a0c5f4bec24583dab2dd50adcea51" Oct 30 02:15:19.976128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.976108 5035 scope.go:122] "RemoveContainer" containerID="d3d5fd47e217cca50bd5e18e4b7d34243042775386b14673f0f12f9871f16518" Oct 30 02:15:19.986872 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:19.986849 5035 scope.go:122] "RemoveContainer" containerID="e269252d340bcf2b12f8c346ef0af07b491ce3d6f17a6bbbac9823c810c2f990" Oct 30 02:15:20.010239 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:20.010194 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm"] Oct 30 02:15:20.023633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:20.023598 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888775-rzhgm"] Oct 30 02:15:20.050497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:20.050457 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb"] Oct 30 02:15:20.062571 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:20.062522 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3a00b8-29888775-h2xpb"] Oct 30 02:15:20.098189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:20.098153 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw"] Oct 30 02:15:20.110995 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:20.110959 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4153e5-29888775-2pgrw"] Oct 30 02:15:20.147255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:20.147207 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6"] Oct 30 02:15:20.153804 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:20.153769 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888775-rs6m6"] Oct 30 02:15:21.645232 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:21.645196 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" path="/var/lib/kubelet/pods/429b5fb1-4e22-4e6d-ae49-0f0fd1972095/volumes" Oct 30 02:15:21.645701 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:21.645687 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" path="/var/lib/kubelet/pods/459172b3-dfd3-46c0-9338-7640a59b6d44/volumes" Oct 30 02:15:21.646093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:21.646081 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" path="/var/lib/kubelet/pods/4ea2f6d5-3cd2-4d6f-806e-de5e3f560319/volumes" Oct 30 02:15:21.646825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:21.646810 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99be1c56-597c-4776-b219-44b73eaebfae" path="/var/lib/kubelet/pods/99be1c56-597c-4776-b219-44b73eaebfae/volumes" Oct 30 02:15:22.858841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:22.858807 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:15:34.833084 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833041 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2"] Oct 30 02:15:34.833880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833863 5035 cpu_manager.go:414] "containerMap: removing container" podUID="99be1c56-597c-4776-b219-44b73eaebfae" containerName="cleanup" Oct 30 02:15:34.833941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833880 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99be1c56-597c-4776-b219-44b73eaebfae" containerName="cleanup" Oct 30 02:15:34.833941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833896 5035 cpu_manager.go:414] "containerMap: removing container" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerName="cleanup" Oct 30 02:15:34.833941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833901 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerName="cleanup" Oct 30 02:15:34.833941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833912 5035 cpu_manager.go:414] "containerMap: removing container" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerName="cleanup" Oct 30 02:15:34.833941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833917 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerName="cleanup" Oct 30 02:15:34.833941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833931 5035 cpu_manager.go:414] "containerMap: removing container" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerName="cleanup" Oct 30 02:15:34.833941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833936 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833950 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833955 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833978 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833984 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.833996 5035 cpu_manager.go:414] "containerMap: removing container" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834001 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834012 5035 cpu_manager.go:414] "containerMap: removing container" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834018 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834027 5035 cpu_manager.go:414] "containerMap: removing container" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834033 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834040 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834045 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834066 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834071 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834078 5035 cpu_manager.go:414] "containerMap: removing container" podUID="99be1c56-597c-4776-b219-44b73eaebfae" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834083 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99be1c56-597c-4776-b219-44b73eaebfae" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834102 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834107 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834115 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerName="cleanup" Oct 30 02:15:34.834210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834120 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834227 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834238 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834245 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834254 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834259 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="99be1c56-597c-4776-b219-44b73eaebfae" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834266 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834273 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1884eb8f-8045-44a4-908c-13bbc9995d64" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834281 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834286 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834293 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834300 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834307 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834314 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="99be1c56-597c-4776-b219-44b73eaebfae" containerName="cleanup" Oct 30 02:15:34.834857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.834319 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerName="cleanup" Oct 30 02:15:34.840512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.840486 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:34.858206 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.858171 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2"] Oct 30 02:15:34.867796 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.867766 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"payload-processing-external-model-plugins-e2e-extmod-b-93ec46\"" Oct 30 02:15:34.869101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.869076 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"routing-overlay\"" Oct 30 02:15:34.869466 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.869442 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"kube-root-ca.crt\"" Oct 30 02:15:34.869618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.869491 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"agc-cred-966beba80607\"" Oct 30 02:15:34.869711 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.869683 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"openshift-service-ca.crt\"" Oct 30 02:15:34.870932 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.870906 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"payload-processing-external-model-e2e--dd78327c-dockercfg-t4cvb\"" Oct 30 02:15:34.934250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.934203 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:34.934477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.934323 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:34.934568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.934470 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r79d\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-kube-api-access-8r79d\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:34.934634 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:34.934610 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.041583 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.038415 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r79d\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-kube-api-access-8r79d\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.041583 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.038520 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.041583 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.038815 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.041583 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.038844 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.041583 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.041032 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.042020 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.041907 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.043792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.043764 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.065483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.065444 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r79d\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-kube-api-access-8r79d\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.156473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.156371 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:35.475077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.475041 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2"] Oct 30 02:15:35.476187 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:35.476160 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3d171c_4f0a_4a67_9226_1e9e6fb58488.slice/crio-421d43e5612773a6f865b7e3f213e0c8a89e26c26235b4624fec9139d868f7ed WatchSource:0}: Error finding container 421d43e5612773a6f865b7e3f213e0c8a89e26c26235b4624fec9139d868f7ed: Status 404 returned error can't find the container with id 421d43e5612773a6f865b7e3f213e0c8a89e26c26235b4624fec9139d868f7ed Oct 30 02:15:35.684429 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.684387 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6"] Oct 30 02:15:35.685292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685273 5035 cpu_manager.go:414] "containerMap: removing container" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerName="cleanup" Oct 30 02:15:35.685292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685293 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerName="cleanup" Oct 30 02:15:35.685425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685315 5035 cpu_manager.go:414] "containerMap: removing container" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerName="cleanup" Oct 30 02:15:35.685425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685320 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459172b3-dfd3-46c0-9338-7640a59b6d44" containerName="cleanup" Oct 30 02:15:35.685425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685374 5035 cpu_manager.go:414] "containerMap: removing container" podUID="99be1c56-597c-4776-b219-44b73eaebfae" containerName="cleanup" Oct 30 02:15:35.685425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685379 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99be1c56-597c-4776-b219-44b73eaebfae" containerName="cleanup" Oct 30 02:15:35.685425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685399 5035 cpu_manager.go:414] "containerMap: removing container" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerName="cleanup" Oct 30 02:15:35.685425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685405 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerName="cleanup" Oct 30 02:15:35.685663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685516 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea2f6d5-3cd2-4d6f-806e-de5e3f560319" containerName="cleanup" Oct 30 02:15:35.685663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685544 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="429b5fb1-4e22-4e6d-ae49-0f0fd1972095" containerName="cleanup" Oct 30 02:15:35.685663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685562 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8f3eda-f86d-4b97-adb7-131d606bf442" containerName="cleanup" Oct 30 02:15:35.685663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.685573 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="99be1c56-597c-4776-b219-44b73eaebfae" containerName="cleanup" Oct 30 02:15:35.690125 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.690095 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:35.707051 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.707010 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9bf0e2\"" Oct 30 02:15:35.710593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.710561 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6"] Oct 30 02:15:35.719546 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.719502 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-9bf0e2-dockercfg-v68lp\"" Oct 30 02:15:35.751505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.750510 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjwk\" (UniqueName: \"kubernetes.io/projected/50eaf685-72df-4e61-afde-a72a20aa72aa-kube-api-access-4rjwk\") pod \"payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6\" (UID: \"50eaf685-72df-4e61-afde-a72a20aa72aa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:35.751505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.750954 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaf685-72df-4e61-afde-a72a20aa72aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6\" (UID: \"50eaf685-72df-4e61-afde-a72a20aa72aa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:35.862036 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.861892 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaf685-72df-4e61-afde-a72a20aa72aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6\" (UID: \"50eaf685-72df-4e61-afde-a72a20aa72aa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:35.873348 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.862589 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjwk\" (UniqueName: \"kubernetes.io/projected/50eaf685-72df-4e61-afde-a72a20aa72aa-kube-api-access-4rjwk\") pod \"payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6\" (UID: \"50eaf685-72df-4e61-afde-a72a20aa72aa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:35.873348 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.864135 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaf685-72df-4e61-afde-a72a20aa72aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6\" (UID: \"50eaf685-72df-4e61-afde-a72a20aa72aa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:35.873348 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.864196 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6"] Oct 30 02:15:35.875755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.875718 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6"] Oct 30 02:15:35.875919 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.875841 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:35.898540 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.898489 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjwk\" (UniqueName: \"kubernetes.io/projected/50eaf685-72df-4e61-afde-a72a20aa72aa-kube-api-access-4rjwk\") pod \"payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6\" (UID: \"50eaf685-72df-4e61-afde-a72a20aa72aa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:35.964758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.964698 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6\" (UID: \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\") " pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:35.964993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:35.964852 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnndd\" (UniqueName: \"kubernetes.io/projected/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-kube-api-access-rnndd\") pod \"payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6\" (UID: \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\") " pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:36.013205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.013116 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:36.066793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.066291 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6\" (UID: \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\") " pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:36.066793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.066376 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnndd\" (UniqueName: \"kubernetes.io/projected/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-kube-api-access-rnndd\") pod \"payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6\" (UID: \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\") " pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:36.067158 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.067133 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6\" (UID: \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\") " pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:36.101511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.092152 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnndd\" (UniqueName: \"kubernetes.io/projected/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-kube-api-access-rnndd\") pod \"payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6\" (UID: \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\") " pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:36.126562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.126483 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" event={"ID":"6b3d171c-4f0a-4a67-9226-1e9e6fb58488","Type":"ContainerStarted","Data":"3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e"} Oct 30 02:15:36.126562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.126562 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" event={"ID":"6b3d171c-4f0a-4a67-9226-1e9e6fb58488","Type":"ContainerStarted","Data":"421d43e5612773a6f865b7e3f213e0c8a89e26c26235b4624fec9139d868f7ed"} Oct 30 02:15:36.126835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.126592 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:36.175746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.175676 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" podStartSLOduration=2.175659132 podStartE2EDuration="2.175659132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:36.171151551 +0000 UTC m=+2814.674083771" watchObservedRunningTime="2026-10-30 02:15:36.175659132 +0000 UTC m=+2814.678591394" Oct 30 02:15:36.200880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.200823 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:36.340474 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.340437 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6"] Oct 30 02:15:36.344369 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:36.344330 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eaf685_72df_4e61_afde_a72a20aa72aa.slice/crio-43a31efd0181755e77616e3bac1e6dcb3d10cf4eeaf612cf71bd522b152df964 WatchSource:0}: Error finding container 43a31efd0181755e77616e3bac1e6dcb3d10cf4eeaf612cf71bd522b152df964: Status 404 returned error can't find the container with id 43a31efd0181755e77616e3bac1e6dcb3d10cf4eeaf612cf71bd522b152df964 Oct 30 02:15:36.514301 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:36.514271 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6"] Oct 30 02:15:36.515157 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:36.515100 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dd380e_2611_46c6_8bd8_6f8ae05cbc33.slice/crio-67a8029c3236c605b21c6f3d0cb3593cca1a021ec3da88bc16392b12091778c0 WatchSource:0}: Error finding container 67a8029c3236c605b21c6f3d0cb3593cca1a021ec3da88bc16392b12091778c0: Status 404 returned error can't find the container with id 67a8029c3236c605b21c6f3d0cb3593cca1a021ec3da88bc16392b12091778c0 Oct 30 02:15:37.136267 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:37.136213 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" event={"ID":"50eaf685-72df-4e61-afde-a72a20aa72aa","Type":"ContainerStarted","Data":"77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806"} Oct 30 02:15:37.136781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:37.136291 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" event={"ID":"50eaf685-72df-4e61-afde-a72a20aa72aa","Type":"ContainerStarted","Data":"43a31efd0181755e77616e3bac1e6dcb3d10cf4eeaf612cf71bd522b152df964"} Oct 30 02:15:37.136781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:37.136457 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:37.138231 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:37.138199 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" event={"ID":"89dd380e-2611-46c6-8bd8-6f8ae05cbc33","Type":"ContainerStarted","Data":"d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a"} Oct 30 02:15:37.138391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:37.138254 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" event={"ID":"89dd380e-2611-46c6-8bd8-6f8ae05cbc33","Type":"ContainerStarted","Data":"67a8029c3236c605b21c6f3d0cb3593cca1a021ec3da88bc16392b12091778c0"} Oct 30 02:15:37.138391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:37.138284 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:37.182248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:37.182176 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" podStartSLOduration=2.18215074 podStartE2EDuration="2.18215074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:37.179944416 +0000 UTC m=+2815.682876658" watchObservedRunningTime="2026-10-30 02:15:37.18215074 +0000 UTC m=+2815.685082958" Oct 30 02:15:37.226775 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:37.226718 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" podStartSLOduration=2.226701941 podStartE2EDuration="2.226701941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:37.223678736 +0000 UTC m=+2815.726610958" watchObservedRunningTime="2026-10-30 02:15:37.226701941 +0000 UTC m=+2815.729634206" Oct 30 02:15:47.142319 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:47.142283 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:15:47.630599 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:15:47.630520 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:15:48.150089 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:48.150056 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:15:48.150633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:48.150142 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:15:49.253552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.253496 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4"] Oct 30 02:15:49.260096 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.260062 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.283927 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.283891 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0ce950-serving-cert\"" Oct 30 02:15:49.284352 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.284326 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4"] Oct 30 02:15:49.383236 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.383191 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-metrics-certs\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.383442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.383321 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r64j\" (UniqueName: \"kubernetes.io/projected/46681de3-fb8d-498e-a12f-2a2eee5f5a40-kube-api-access-7r64j\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.383442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.383372 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46681de3-fb8d-498e-a12f-2a2eee5f5a40-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.383442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.383432 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.484573 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.484490 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-metrics-certs\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.484787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.484667 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r64j\" (UniqueName: \"kubernetes.io/projected/46681de3-fb8d-498e-a12f-2a2eee5f5a40-kube-api-access-7r64j\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.484787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.484720 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46681de3-fb8d-498e-a12f-2a2eee5f5a40-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.484914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.484798 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.485424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.485399 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46681de3-fb8d-498e-a12f-2a2eee5f5a40-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.487209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.487176 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-metrics-certs\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.487350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.487334 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.541081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.540971 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r64j\" (UniqueName: \"kubernetes.io/projected/46681de3-fb8d-498e-a12f-2a2eee5f5a40-kube-api-access-7r64j\") pod \"maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.579120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.579080 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:49.896701 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:49.896665 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4"] Oct 30 02:15:49.897265 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:15:49.897238 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46681de3_fb8d_498e_a12f_2a2eee5f5a40.slice/crio-6d2137e6cd1fd84833e1ba7b59b0da5fcaa7d3920d7c11376dc1dc8e91a2a9c3 WatchSource:0}: Error finding container 6d2137e6cd1fd84833e1ba7b59b0da5fcaa7d3920d7c11376dc1dc8e91a2a9c3: Status 404 returned error can't find the container with id 6d2137e6cd1fd84833e1ba7b59b0da5fcaa7d3920d7c11376dc1dc8e91a2a9c3 Oct 30 02:15:50.246055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:50.246015 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" event={"ID":"46681de3-fb8d-498e-a12f-2a2eee5f5a40","Type":"ContainerStarted","Data":"6d2137e6cd1fd84833e1ba7b59b0da5fcaa7d3920d7c11376dc1dc8e91a2a9c3"} Oct 30 02:15:51.255840 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:51.255799 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" event={"ID":"46681de3-fb8d-498e-a12f-2a2eee5f5a40","Type":"ContainerStarted","Data":"3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c"} Oct 30 02:15:51.256340 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:51.255870 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:15:51.303635 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:51.303518 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" podStartSLOduration=1.965023245 podStartE2EDuration="2.303503382s" totalImagesPullingTime="338.480137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:51.302861861 +0000 UTC m=+2829.805794081" watchObservedRunningTime="2026-10-30 02:15:51.303503382 +0000 UTC m=+2829.806435601" Oct 30 02:15:57.271314 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:15:57.271232 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:16:17.631275 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:16:17.631230 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:16:32.884309 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:32.884269 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh"] Oct 30 02:16:32.884842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:32.884516 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" containerID="cri-o://a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218" gracePeriod=30 Oct 30 02:16:36.971517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:36.971471 5035 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:16:36.972001 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:36.971584 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:16:38.174494 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.174416 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 02:16:38.187617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.187583 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-token\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-token\") pod \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " Oct 30 02:16:38.187761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.187679 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-envoy\") pod \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " Oct 30 02:16:38.187806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.187772 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-data\") pod \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " Oct 30 02:16:38.187867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.187850 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istiod-ca-cert\") pod \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " Oct 30 02:16:38.187941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.187923 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-socket\") pod \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " Oct 30 02:16:38.188003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.187993 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-podinfo\") pod \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " Oct 30 02:16:38.188061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188051 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-certs\") pod \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " Oct 30 02:16:38.188453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188103 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-data" pod "0070f21f-823c-45ab-8264-39ba6c3f3c9e" (UID: "0070f21f-823c-45ab-8264-39ba6c3f3c9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:38.188453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188240 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-credential-socket\") pod \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " Oct 30 02:16:38.188453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188284 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-kube-api-access-rz4r8\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-kube-api-access-rz4r8\") pod \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\" (UID: \"0070f21f-823c-45ab-8264-39ba6c3f3c9e\") " Oct 30 02:16:38.188453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188287 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istiod-ca-cert" pod "0070f21f-823c-45ab-8264-39ba6c3f3c9e" (UID: "0070f21f-823c-45ab-8264-39ba6c3f3c9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:38.188453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188419 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-certs" pod "0070f21f-823c-45ab-8264-39ba6c3f3c9e" (UID: "0070f21f-823c-45ab-8264-39ba6c3f3c9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:38.189018 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188880 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-socket" pod "0070f21f-823c-45ab-8264-39ba6c3f3c9e" (UID: "0070f21f-823c-45ab-8264-39ba6c3f3c9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:38.189018 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188905 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.189018 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188928 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.189018 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188943 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.189018 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.188983 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-credential-socket" pod "0070f21f-823c-45ab-8264-39ba6c3f3c9e" (UID: "0070f21f-823c-45ab-8264-39ba6c3f3c9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:38.190325 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.190281 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-token" pod "0070f21f-823c-45ab-8264-39ba6c3f3c9e" (UID: "0070f21f-823c-45ab-8264-39ba6c3f3c9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:38.190563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.190507 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-envoy" pod "0070f21f-823c-45ab-8264-39ba6c3f3c9e" (UID: "0070f21f-823c-45ab-8264-39ba6c3f3c9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:38.190762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.190740 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-podinfo" pod "0070f21f-823c-45ab-8264-39ba6c3f3c9e" (UID: "0070f21f-823c-45ab-8264-39ba6c3f3c9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:16:38.191058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.191034 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-kube-api-access-rz4r8" pod "0070f21f-823c-45ab-8264-39ba6c3f3c9e" (UID: "0070f21f-823c-45ab-8264-39ba6c3f3c9e"). InnerVolumeSpecName "kube-api-access-rz4r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:38.290917 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.290870 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.291120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.290938 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.291120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.290948 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.291120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.290959 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4r8\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-kube-api-access-rz4r8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.291120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.290969 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.291120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.290977 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0070f21f-823c-45ab-8264-39ba6c3f3c9e-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.665956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.665920 5035 generic.go:356] "Generic (PLEG): container finished" podID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerID="a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218" exitCode=0 Oct 30 02:16:38.666262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.665984 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" Oct 30 02:16:38.666262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.665988 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" event={"ID":"0070f21f-823c-45ab-8264-39ba6c3f3c9e","Type":"ContainerDied","Data":"a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218"} Oct 30 02:16:38.666262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.666114 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh" event={"ID":"0070f21f-823c-45ab-8264-39ba6c3f3c9e","Type":"ContainerDied","Data":"f9f1569934dfb7e7d62450570cc1732bdc6bb5365dbec25f29439119da769f9f"} Oct 30 02:16:38.666262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.666138 5035 scope.go:122] "RemoveContainer" containerID="a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218" Oct 30 02:16:38.680862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.680838 5035 scope.go:122] "RemoveContainer" containerID="a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218" Oct 30 02:16:38.681190 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:16:38.681164 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218\": container with ID starting with a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218 not found: ID does not exist" containerID="a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218" Oct 30 02:16:38.681275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.681200 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218"} err="failed to get container status \"a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218\": rpc error: code = NotFound desc = could not find container \"a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218\": container with ID starting with a0d20ac43e9ba61afc54ec568a978ccc2c21ed5d7505bbb71274e51f02a06218 not found: ID does not exist" Oct 30 02:16:38.725855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.725821 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh"] Oct 30 02:16:38.739435 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:38.739397 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-fqmxh"] Oct 30 02:16:39.645002 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:39.644970 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" path="/var/lib/kubelet/pods/0070f21f-823c-45ab-8264-39ba6c3f3c9e/volumes" Oct 30 02:16:41.782177 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.782137 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l"] Oct 30 02:16:41.782978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.782960 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" Oct 30 02:16:41.783029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.782980 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" Oct 30 02:16:41.783101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.783091 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0070f21f-823c-45ab-8264-39ba6c3f3c9e" containerName="istio-proxy" Oct 30 02:16:41.786850 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.786824 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:16:41.806720 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.806686 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l"] Oct 30 02:16:41.815985 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.815951 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-000edd\"" Oct 30 02:16:41.815985 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.815954 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-000edd-dockercfg-7w85t\"" Oct 30 02:16:41.829185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.829147 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq"] Oct 30 02:16:41.833892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.833863 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:41.840305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.840266 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l\" (UID: \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:16:41.840791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.840766 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rzn\" (UniqueName: \"kubernetes.io/projected/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-kube-api-access-j9rzn\") pod \"payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l\" (UID: \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:16:41.871603 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.871557 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq"] Oct 30 02:16:41.954590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.943426 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvw2t\" (UniqueName: \"kubernetes.io/projected/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-kube-api-access-bvw2t\") pod \"payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq\" (UID: \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:41.954590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.943495 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l\" (UID: \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:16:41.954590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.943705 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq\" (UID: \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:41.954590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.943762 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rzn\" (UniqueName: \"kubernetes.io/projected/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-kube-api-access-j9rzn\") pod \"payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l\" (UID: \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:16:41.954590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:41.945171 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l\" (UID: \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:16:42.016552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.011760 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rzn\" (UniqueName: \"kubernetes.io/projected/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-kube-api-access-j9rzn\") pod \"payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l\" (UID: \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:16:42.044845 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.044743 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq\" (UID: \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:42.045046 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.044911 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvw2t\" (UniqueName: \"kubernetes.io/projected/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-kube-api-access-bvw2t\") pod \"payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq\" (UID: \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:42.045624 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.045595 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq\" (UID: \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:42.070336 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.070298 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvw2t\" (UniqueName: \"kubernetes.io/projected/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-kube-api-access-bvw2t\") pod \"payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq\" (UID: \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:42.103450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.103407 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:16:42.162696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.162656 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:42.426419 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:16:42.426387 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0632efb9_a10a_4f96_8261_2b6ca3e1ffdf.slice/crio-df1f4387f3cee0aab3bda8ca78b1cb094c7997ae0ec079351503da229f74d22b WatchSource:0}: Error finding container df1f4387f3cee0aab3bda8ca78b1cb094c7997ae0ec079351503da229f74d22b: Status 404 returned error can't find the container with id df1f4387f3cee0aab3bda8ca78b1cb094c7997ae0ec079351503da229f74d22b Oct 30 02:16:42.427053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.427020 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l"] Oct 30 02:16:42.518611 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.518578 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq"] Oct 30 02:16:42.519855 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:16:42.519826 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c9feb1_42a4_4fa3_8b30_fe3ed5aa69c0.slice/crio-08d28101a5116481b15896c9ba399222dafdf1a8b2993c85663a41386660cea0 WatchSource:0}: Error finding container 08d28101a5116481b15896c9ba399222dafdf1a8b2993c85663a41386660cea0: Status 404 returned error can't find the container with id 08d28101a5116481b15896c9ba399222dafdf1a8b2993c85663a41386660cea0 Oct 30 02:16:42.700833 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.700791 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" event={"ID":"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf","Type":"ContainerStarted","Data":"df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116"} Oct 30 02:16:42.701017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.700865 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" event={"ID":"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf","Type":"ContainerStarted","Data":"df1f4387f3cee0aab3bda8ca78b1cb094c7997ae0ec079351503da229f74d22b"} Oct 30 02:16:42.701017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.700894 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:16:42.703093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.703053 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" event={"ID":"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0","Type":"ContainerStarted","Data":"818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee"} Oct 30 02:16:42.703273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.703122 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" event={"ID":"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0","Type":"ContainerStarted","Data":"08d28101a5116481b15896c9ba399222dafdf1a8b2993c85663a41386660cea0"} Oct 30 02:16:42.703273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.703151 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:42.747709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.747652 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" podStartSLOduration=1.747635791 podStartE2EDuration="1.747635791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:16:42.745091188 +0000 UTC m=+2881.248023409" watchObservedRunningTime="2026-10-30 02:16:42.747635791 +0000 UTC m=+2881.250568064" Oct 30 02:16:42.794204 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:42.794139 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" podStartSLOduration=1.794121225 podStartE2EDuration="1.794121225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:16:42.78859548 +0000 UTC m=+2881.291527702" watchObservedRunningTime="2026-10-30 02:16:42.794121225 +0000 UTC m=+2881.297053444" Oct 30 02:16:47.634090 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:16:47.634049 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:16:53.715270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:53.715232 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:16:53.715836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:16:53.715317 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:17:17.631134 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:17:17.631090 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:17:19.104380 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.104339 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb"] Oct 30 02:17:19.110560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.110516 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.131510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.131476 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb"] Oct 30 02:17:19.132031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.131998 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-000edd-serving-cert\"" Oct 30 02:17:19.212907 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.212867 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.213108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.213026 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-maas-api-tls\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.213108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.213052 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-metrics-certs\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.213108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.213091 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzr8b\" (UniqueName: \"kubernetes.io/projected/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-kube-api-access-nzr8b\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.314589 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.314504 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-maas-api-tls\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.314808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.314598 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-metrics-certs\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.314808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.314652 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzr8b\" (UniqueName: \"kubernetes.io/projected/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-kube-api-access-nzr8b\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.315003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.314983 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.315875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.315851 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.318250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.318222 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-metrics-certs\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.318250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.318247 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-maas-api-tls\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.340661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.340622 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzr8b\" (UniqueName: \"kubernetes.io/projected/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-kube-api-access-nzr8b\") pod \"maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.447628 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.447511 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:19.763863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:19.763827 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb"] Oct 30 02:17:19.765706 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:17:19.765677 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dab5d1f_f9ea_4b1f_82cf_97ebbd947993.slice/crio-b882acd11cbe70792d281f1b0dfb27fd850cef6cc7ed6514ba4b5fcce340b8d2 WatchSource:0}: Error finding container b882acd11cbe70792d281f1b0dfb27fd850cef6cc7ed6514ba4b5fcce340b8d2: Status 404 returned error can't find the container with id b882acd11cbe70792d281f1b0dfb27fd850cef6cc7ed6514ba4b5fcce340b8d2 Oct 30 02:17:20.016851 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:20.016760 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" event={"ID":"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993","Type":"ContainerStarted","Data":"b882acd11cbe70792d281f1b0dfb27fd850cef6cc7ed6514ba4b5fcce340b8d2"} Oct 30 02:17:21.025854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:21.025809 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" event={"ID":"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993","Type":"ContainerStarted","Data":"f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91"} Oct 30 02:17:21.026382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:21.025873 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:21.073930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:21.073872 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" podStartSLOduration=1.638669804 podStartE2EDuration="2.073855397s" totalImagesPullingTime="435.185593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:17:21.07139206 +0000 UTC m=+2919.574324279" watchObservedRunningTime="2026-10-30 02:17:21.073855397 +0000 UTC m=+2919.576787617" Oct 30 02:17:27.040277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:27.040196 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:17:35.361231 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:35.361179 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2"] Oct 30 02:17:40.362292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.362246 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" podUID="6b3d171c-4f0a-4a67-9226-1e9e6fb58488" containerName="payload-processing" containerID="cri-o://3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e" gracePeriod=25 Oct 30 02:17:40.806324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.806287 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:17:40.904007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.903321 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-routing-overlay-volume\") pod \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " Oct 30 02:17:40.904007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.903541 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-provider-credentials\") pod \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " Oct 30 02:17:40.904007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.903613 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-plugins-config-volume\") pod \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " Oct 30 02:17:40.904007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.903734 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-kube-api-access-8r79d\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-kube-api-access-8r79d\") pod \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\" (UID: \"6b3d171c-4f0a-4a67-9226-1e9e6fb58488\") " Oct 30 02:17:40.905932 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.905837 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-routing-overlay-volume" pod "6b3d171c-4f0a-4a67-9226-1e9e6fb58488" (UID: "6b3d171c-4f0a-4a67-9226-1e9e6fb58488"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:40.907302 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.907271 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-plugins-config-volume" pod "6b3d171c-4f0a-4a67-9226-1e9e6fb58488" (UID: "6b3d171c-4f0a-4a67-9226-1e9e6fb58488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:40.907466 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.907365 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:17:40.910447 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.910403 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-provider-credentials" pod "6b3d171c-4f0a-4a67-9226-1e9e6fb58488" (UID: "6b3d171c-4f0a-4a67-9226-1e9e6fb58488"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:40.950918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:40.948996 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-kube-api-access-8r79d" pod "6b3d171c-4f0a-4a67-9226-1e9e6fb58488" (UID: "6b3d171c-4f0a-4a67-9226-1e9e6fb58488"). InnerVolumeSpecName "kube-api-access-8r79d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:41.009222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.009185 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:17:41.009222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.009221 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:17:41.009467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.009234 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r79d\" (UniqueName: \"kubernetes.io/projected/6b3d171c-4f0a-4a67-9226-1e9e6fb58488-kube-api-access-8r79d\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:17:41.206901 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.206802 5035 generic.go:356] "Generic (PLEG): container finished" podID="6b3d171c-4f0a-4a67-9226-1e9e6fb58488" containerID="3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e" exitCode=0 Oct 30 02:17:41.206901 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.206864 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" Oct 30 02:17:41.206901 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.206885 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" event={"ID":"6b3d171c-4f0a-4a67-9226-1e9e6fb58488","Type":"ContainerDied","Data":"3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e"} Oct 30 02:17:41.207160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.206953 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2" event={"ID":"6b3d171c-4f0a-4a67-9226-1e9e6fb58488","Type":"ContainerDied","Data":"421d43e5612773a6f865b7e3f213e0c8a89e26c26235b4624fec9139d868f7ed"} Oct 30 02:17:41.207160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.206984 5035 scope.go:122] "RemoveContainer" containerID="3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e" Oct 30 02:17:41.219407 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.219382 5035 scope.go:122] "RemoveContainer" containerID="3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e" Oct 30 02:17:41.219785 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:17:41.219754 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e\": container with ID starting with 3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e not found: ID does not exist" containerID="3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e" Oct 30 02:17:41.219883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.219792 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e"} err="failed to get container status \"3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e\": rpc error: code = NotFound desc = could not find container \"3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e\": container with ID starting with 3d42c833f316cc26e26b5f955bd35ac49b7a464a21021f077e91656a9a44023e not found: ID does not exist" Oct 30 02:17:41.266067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.266032 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2"] Oct 30 02:17:41.280419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.280380 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-598dzcqx2"] Oct 30 02:17:41.646385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:41.646304 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3d171c-4f0a-4a67-9226-1e9e6fb58488" path="/var/lib/kubelet/pods/6b3d171c-4f0a-4a67-9226-1e9e6fb58488/volumes" Oct 30 02:17:47.632231 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:17:47.632187 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:17:51.657788 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.657751 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2"] Oct 30 02:17:51.658624 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.658607 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6b3d171c-4f0a-4a67-9226-1e9e6fb58488" containerName="payload-processing" Oct 30 02:17:51.658683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.658625 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3d171c-4f0a-4a67-9226-1e9e6fb58488" containerName="payload-processing" Oct 30 02:17:51.658769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.658758 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3d171c-4f0a-4a67-9226-1e9e6fb58488" containerName="payload-processing" Oct 30 02:17:51.664341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.664311 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.690255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.687925 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"routing-overlay\"" Oct 30 02:17:51.690255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.688077 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"openshift-service-ca.crt\"" Oct 30 02:17:51.690255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.688143 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-58fad4\"" Oct 30 02:17:51.690255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.692863 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2"] Oct 30 02:17:51.690255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.695722 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.690255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.695810 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.690255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.695856 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4scm\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-kube-api-access-c4scm\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.690255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.695999 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.697963 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.697935 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"e2e-cmp-legacy-95eb0f-provider-cred\"" Oct 30 02:17:51.698110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.698041 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"kube-root-ca.crt\"" Oct 30 02:17:51.698181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.698122 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"payload-processing-external-model-e2e--4b9246f1-dockercfg-chwbf\"" Oct 30 02:17:51.801799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.801675 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.802568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.802143 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.802568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.802207 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4scm\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-kube-api-access-c4scm\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.802568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.802316 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.803975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.803947 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.805457 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.805429 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.811925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.811857 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.846554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.845230 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4scm\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-kube-api-access-c4scm\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:51.980639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:51.980600 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:52.295201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:52.295169 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2"] Oct 30 02:17:52.297002 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:17:52.296972 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8320748_e22f_473d_a50e_e3f877e385b1.slice/crio-744160006a0da82f1c16e5ede5c54672b836af4db358f127bafd31b523533c90 WatchSource:0}: Error finding container 744160006a0da82f1c16e5ede5c54672b836af4db358f127bafd31b523533c90: Status 404 returned error can't find the container with id 744160006a0da82f1c16e5ede5c54672b836af4db358f127bafd31b523533c90 Oct 30 02:17:53.309691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:53.309649 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" event={"ID":"f8320748-e22f-473d-a50e-e3f877e385b1","Type":"ContainerStarted","Data":"1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83"} Oct 30 02:17:53.310146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:53.309744 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" event={"ID":"f8320748-e22f-473d-a50e-e3f877e385b1","Type":"ContainerStarted","Data":"744160006a0da82f1c16e5ede5c54672b836af4db358f127bafd31b523533c90"} Oct 30 02:17:53.310146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:53.309847 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:17:53.356641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:17:53.356577 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" podStartSLOduration=2.356559792 podStartE2EDuration="2.356559792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:17:53.354890224 +0000 UTC m=+2951.857822444" watchObservedRunningTime="2026-10-30 02:17:53.356559792 +0000 UTC m=+2951.859492011" Oct 30 02:18:04.321270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:04.321237 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:18:04.401170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:04.401130 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6"] Oct 30 02:18:04.439466 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:04.439429 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6"] Oct 30 02:18:09.402501 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.402432 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" podUID="89dd380e-2611-46c6-8bd8-6f8ae05cbc33" containerName="payload-processing" containerID="cri-o://d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a" gracePeriod=25 Oct 30 02:18:09.439853 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.439812 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" podUID="50eaf685-72df-4e61-afde-a72a20aa72aa" containerName="payload-pre-processing" containerID="cri-o://77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806" gracePeriod=25 Oct 30 02:18:09.780228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.778856 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:18:09.825046 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.825005 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-kube-api-access-rnndd\" (UniqueName: \"kubernetes.io/projected/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-kube-api-access-rnndd\") pod \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\" (UID: \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\") " Oct 30 02:18:09.825346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.825331 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-plugins-config-volume\") pod \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\" (UID: \"89dd380e-2611-46c6-8bd8-6f8ae05cbc33\") " Oct 30 02:18:09.826116 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.826087 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-plugins-config-volume" pod "89dd380e-2611-46c6-8bd8-6f8ae05cbc33" (UID: "89dd380e-2611-46c6-8bd8-6f8ae05cbc33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:09.849275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.841711 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-kube-api-access-rnndd" pod "89dd380e-2611-46c6-8bd8-6f8ae05cbc33" (UID: "89dd380e-2611-46c6-8bd8-6f8ae05cbc33"). InnerVolumeSpecName "kube-api-access-rnndd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:09.865661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.865617 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:18:09.926570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.926453 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50eaf685-72df-4e61-afde-a72a20aa72aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaf685-72df-4e61-afde-a72a20aa72aa-plugins-config-volume\") pod \"50eaf685-72df-4e61-afde-a72a20aa72aa\" (UID: \"50eaf685-72df-4e61-afde-a72a20aa72aa\") " Oct 30 02:18:09.926842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.926617 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50eaf685-72df-4e61-afde-a72a20aa72aa-kube-api-access-4rjwk\" (UniqueName: \"kubernetes.io/projected/50eaf685-72df-4e61-afde-a72a20aa72aa-kube-api-access-4rjwk\") pod \"50eaf685-72df-4e61-afde-a72a20aa72aa\" (UID: \"50eaf685-72df-4e61-afde-a72a20aa72aa\") " Oct 30 02:18:09.927601 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.927569 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50eaf685-72df-4e61-afde-a72a20aa72aa-plugins-config-volume" pod "50eaf685-72df-4e61-afde-a72a20aa72aa" (UID: "50eaf685-72df-4e61-afde-a72a20aa72aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:09.927820 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.927577 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:18:09.927989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.927975 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnndd\" (UniqueName: \"kubernetes.io/projected/89dd380e-2611-46c6-8bd8-6f8ae05cbc33-kube-api-access-rnndd\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:18:09.931122 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:09.931085 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eaf685-72df-4e61-afde-a72a20aa72aa-kube-api-access-4rjwk" pod "50eaf685-72df-4e61-afde-a72a20aa72aa" (UID: "50eaf685-72df-4e61-afde-a72a20aa72aa"). InnerVolumeSpecName "kube-api-access-4rjwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:10.029502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.029457 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaf685-72df-4e61-afde-a72a20aa72aa-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:18:10.029502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.029493 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rjwk\" (UniqueName: \"kubernetes.io/projected/50eaf685-72df-4e61-afde-a72a20aa72aa-kube-api-access-4rjwk\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:18:10.469359 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.469320 5035 generic.go:356] "Generic (PLEG): container finished" podID="89dd380e-2611-46c6-8bd8-6f8ae05cbc33" containerID="d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a" exitCode=0 Oct 30 02:18:10.469904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.469367 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" Oct 30 02:18:10.469904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.469401 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" event={"ID":"89dd380e-2611-46c6-8bd8-6f8ae05cbc33","Type":"ContainerDied","Data":"d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a"} Oct 30 02:18:10.469904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.469465 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6" event={"ID":"89dd380e-2611-46c6-8bd8-6f8ae05cbc33","Type":"ContainerDied","Data":"67a8029c3236c605b21c6f3d0cb3593cca1a021ec3da88bc16392b12091778c0"} Oct 30 02:18:10.469904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.469498 5035 scope.go:122] "RemoveContainer" containerID="d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a" Oct 30 02:18:10.471270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.471241 5035 generic.go:356] "Generic (PLEG): container finished" podID="50eaf685-72df-4e61-afde-a72a20aa72aa" containerID="77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806" exitCode=0 Oct 30 02:18:10.471443 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.471296 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" Oct 30 02:18:10.471443 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.471311 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" event={"ID":"50eaf685-72df-4e61-afde-a72a20aa72aa","Type":"ContainerDied","Data":"77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806"} Oct 30 02:18:10.471443 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.471358 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6" event={"ID":"50eaf685-72df-4e61-afde-a72a20aa72aa","Type":"ContainerDied","Data":"43a31efd0181755e77616e3bac1e6dcb3d10cf4eeaf612cf71bd522b152df964"} Oct 30 02:18:10.486554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.486496 5035 scope.go:122] "RemoveContainer" containerID="d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a" Oct 30 02:18:10.486963 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:18:10.486941 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a\": container with ID starting with d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a not found: ID does not exist" containerID="d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a" Oct 30 02:18:10.487039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.486972 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a"} err="failed to get container status \"d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a\": rpc error: code = NotFound desc = could not find container \"d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a\": container with ID starting with d611e8a1f51920f9a9d52ea395b33e491c9076cf212e0885081cf1e24b1a459a not found: ID does not exist" Oct 30 02:18:10.487039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.486992 5035 scope.go:122] "RemoveContainer" containerID="77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806" Oct 30 02:18:10.497967 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.497943 5035 scope.go:122] "RemoveContainer" containerID="77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806" Oct 30 02:18:10.498292 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:18:10.498258 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806\": container with ID starting with 77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806 not found: ID does not exist" containerID="77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806" Oct 30 02:18:10.498378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.498304 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806"} err="failed to get container status \"77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806\": rpc error: code = NotFound desc = could not find container \"77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806\": container with ID starting with 77dcb401867fea7e4542981656d477627747c21dc93f3ed4711d6a05e1f98806 not found: ID does not exist" Oct 30 02:18:10.528961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.528920 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6"] Oct 30 02:18:10.541929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.541889 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9bf0e2-57b5f85fdc-g88g6"] Oct 30 02:18:10.569011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.568972 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6"] Oct 30 02:18:10.572931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:10.572888 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9bf0e2-7dd576c498-rlxz6"] Oct 30 02:18:11.301054 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.300981 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx"] Oct 30 02:18:11.303302 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.303259 5035 cpu_manager.go:414] "containerMap: removing container" podUID="50eaf685-72df-4e61-afde-a72a20aa72aa" containerName="payload-pre-processing" Oct 30 02:18:11.303522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.303503 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50eaf685-72df-4e61-afde-a72a20aa72aa" containerName="payload-pre-processing" Oct 30 02:18:11.303716 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.303703 5035 cpu_manager.go:414] "containerMap: removing container" podUID="89dd380e-2611-46c6-8bd8-6f8ae05cbc33" containerName="payload-processing" Oct 30 02:18:11.303800 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.303789 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89dd380e-2611-46c6-8bd8-6f8ae05cbc33" containerName="payload-processing" Oct 30 02:18:11.304310 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.304283 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="50eaf685-72df-4e61-afde-a72a20aa72aa" containerName="payload-pre-processing" Oct 30 02:18:11.304483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.304471 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="89dd380e-2611-46c6-8bd8-6f8ae05cbc33" containerName="payload-processing" Oct 30 02:18:11.313838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.313804 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.325711 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.325673 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx"] Oct 30 02:18:11.334576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.334141 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"agc-cred-eba0f971cd7a\"" Oct 30 02:18:11.334576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.334179 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"payload-processing-external-model-plugins-e2e-extmod-b-000edd\"" Oct 30 02:18:11.334576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.334227 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"routing-overlay\"" Oct 30 02:18:11.334576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.334254 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"openshift-service-ca.crt\"" Oct 30 02:18:11.344009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.343971 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"kube-root-ca.crt\"" Oct 30 02:18:11.344009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.343971 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"payload-processing-external-model-e2e--74e737ad-dockercfg-bs8rz\"" Oct 30 02:18:11.451518 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.451476 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.451779 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.451660 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.451839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.451781 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqwjj\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-kube-api-access-sqwjj\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.451839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.451830 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.553740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.553643 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqwjj\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-kube-api-access-sqwjj\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.554181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.553805 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.554181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.553937 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.554181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.554065 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.554674 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.554638 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.554872 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.554844 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.556775 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.556747 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.605812 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.605769 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqwjj\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-kube-api-access-sqwjj\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.639949 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.639905 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:11.684478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.682886 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50eaf685-72df-4e61-afde-a72a20aa72aa" path="/var/lib/kubelet/pods/50eaf685-72df-4e61-afde-a72a20aa72aa/volumes" Oct 30 02:18:11.685118 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:11.685091 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89dd380e-2611-46c6-8bd8-6f8ae05cbc33" path="/var/lib/kubelet/pods/89dd380e-2611-46c6-8bd8-6f8ae05cbc33/volumes" Oct 30 02:18:12.005382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:12.005350 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx"] Oct 30 02:18:12.006300 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:18:12.006273 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094e006f_b549_4050_a3b0_5b19d663bd4d.slice/crio-46a9d4140fd3450d57ed2a0566113a7bd5b25139512fe3f3a478e4592523760a WatchSource:0}: Error finding container 46a9d4140fd3450d57ed2a0566113a7bd5b25139512fe3f3a478e4592523760a: Status 404 returned error can't find the container with id 46a9d4140fd3450d57ed2a0566113a7bd5b25139512fe3f3a478e4592523760a Oct 30 02:18:12.491335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:12.491294 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" event={"ID":"094e006f-b549-4050-a3b0-5b19d663bd4d","Type":"ContainerStarted","Data":"ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9"} Oct 30 02:18:12.491632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:12.491366 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" event={"ID":"094e006f-b549-4050-a3b0-5b19d663bd4d","Type":"ContainerStarted","Data":"46a9d4140fd3450d57ed2a0566113a7bd5b25139512fe3f3a478e4592523760a"} Oct 30 02:18:12.491632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:12.491397 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:12.539678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:12.539619 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" podStartSLOduration=1.539602363 podStartE2EDuration="1.539602363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:12.537777109 +0000 UTC m=+2971.040709330" watchObservedRunningTime="2026-10-30 02:18:12.539602363 +0000 UTC m=+2971.042534643" Oct 30 02:18:17.631593 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:18:17.631549 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:18:23.502842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:23.502810 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:18:37.310503 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.310459 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58"] Oct 30 02:18:37.317187 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.317159 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.338697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.338653 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-58fad4-openshift-default-dockercfg-9twt4\"" Oct 30 02:18:37.358148 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.358113 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58"] Oct 30 02:18:37.403467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.403429 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.403703 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.403492 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzwxx\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-kube-api-access-mzwxx\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.403703 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.403621 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-data\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.403840 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.403721 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-credential-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.403840 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.403815 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-podinfo\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.403965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.403880 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-certs\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.404009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.403961 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-envoy\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.404066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.404010 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60bc0-359c-4074-a4ff-7f561627df13-istiod-ca-cert\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.404142 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.404123 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-token\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.505288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505241 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-data\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.505460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505343 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-credential-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.505460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505369 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-podinfo\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.505460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505398 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-certs\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.505460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505436 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-envoy\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.505719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505478 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60bc0-359c-4074-a4ff-7f561627df13-istiod-ca-cert\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.505719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505539 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-token\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.505719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505612 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.505719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505708 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzwxx\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-kube-api-access-mzwxx\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.506013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505780 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-data\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.506013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505834 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-credential-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.506013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.505900 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-certs\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.506194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.506092 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.506405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.506385 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60bc0-359c-4074-a4ff-7f561627df13-istiod-ca-cert\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.507942 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.507918 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-envoy\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.508017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.507954 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-podinfo\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.530549 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.530502 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-token\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.532087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.532058 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzwxx\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-kube-api-access-mzwxx\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.640646 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.634514 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:37.973411 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.973373 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58"] Oct 30 02:18:37.977057 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:18:37.977023 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b60bc0_359c_4074_a4ff_7f561627df13.slice/crio-fa77238dbcdbdb705fd4f89fc974d63d6e3ebe9786520eeb0cfc19cac32680b3 WatchSource:0}: Error finding container fa77238dbcdbdb705fd4f89fc974d63d6e3ebe9786520eeb0cfc19cac32680b3: Status 404 returned error can't find the container with id fa77238dbcdbdb705fd4f89fc974d63d6e3ebe9786520eeb0cfc19cac32680b3 Oct 30 02:18:37.980124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.980088 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:18:37.980211 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.980162 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:18:37.980211 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:37.980192 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:18:38.800505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:38.800469 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" event={"ID":"a4b60bc0-359c-4074-a4ff-7f561627df13","Type":"ContainerStarted","Data":"987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6"} Oct 30 02:18:38.800975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:38.800523 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" event={"ID":"a4b60bc0-359c-4074-a4ff-7f561627df13","Type":"ContainerStarted","Data":"fa77238dbcdbdb705fd4f89fc974d63d6e3ebe9786520eeb0cfc19cac32680b3"} Oct 30 02:18:38.863197 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:38.863124 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" podStartSLOduration=1.863104559 podStartE2EDuration="1.863104559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:38.862027489 +0000 UTC m=+2997.364959701" watchObservedRunningTime="2026-10-30 02:18:38.863104559 +0000 UTC m=+2997.366036780" Oct 30 02:18:39.644196 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:39.644160 5035 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:18:39.644467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:39.644219 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 02:18:39.646407 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:39.646375 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:40.642835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:40.642793 5035 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:18:40.643273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:40.642858 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 02:18:41.642715 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:41.642678 5035 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:18:41.643078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:41.643041 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 02:18:42.642874 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:42.642836 5035 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:18:42.643079 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:42.642905 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 02:18:43.643416 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:43.643372 5035 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:18:43.643884 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:43.643436 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 02:18:43.834920 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:43.834866 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:18:43.850775 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:43.850745 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:18:43.889555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:43.866724 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:18:43.908680 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:43.908596 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:18:44.649173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:44.649138 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:44.649652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:44.649639 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:44.650589 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:18:44.650567 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:18:47.630374 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:18:47.630331 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:19:06.948286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:06.948195 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2"] Oct 30 02:19:06.954724 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:06.954693 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:06.965432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:06.965397 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-metrics-certs\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:06.965653 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:06.965572 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7417f30-4ad3-47c8-835d-b0511309cac5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:06.965736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:06.965663 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:06.965864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:06.965844 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7fq7\" (UniqueName: \"kubernetes.io/projected/e7417f30-4ad3-47c8-835d-b0511309cac5-kube-api-access-r7fq7\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:06.978450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:06.978414 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-95f40f-serving-cert\"" Oct 30 02:19:06.986554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:06.983282 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2"] Oct 30 02:19:07.066737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.066696 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:07.066968 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:19:07.066853 5035 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-serving-cert: secret "maas-api-e2e-extmod-a-95f40f-serving-cert" not found Oct 30 02:19:07.066968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.066895 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7fq7\" (UniqueName: \"kubernetes.io/projected/e7417f30-4ad3-47c8-835d-b0511309cac5-kube-api-access-r7fq7\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:07.066968 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:19:07.066935 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls podName:e7417f30-4ad3-47c8-835d-b0511309cac5 nodeName:}" failed. No retries permitted until 2026-10-30 02:19:07.566919024 +0000 UTC m=+3026.069851222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls") pod "maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" (UID: "e7417f30-4ad3-47c8-835d-b0511309cac5") : secret "maas-api-e2e-extmod-a-95f40f-serving-cert" not found Oct 30 02:19:07.067153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.066983 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-metrics-certs\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:07.067153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.067094 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7417f30-4ad3-47c8-835d-b0511309cac5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:07.067809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.067778 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7417f30-4ad3-47c8-835d-b0511309cac5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:07.069635 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.069611 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-metrics-certs\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:07.092370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.092342 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7fq7\" (UniqueName: \"kubernetes.io/projected/e7417f30-4ad3-47c8-835d-b0511309cac5-kube-api-access-r7fq7\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:07.574921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.574878 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:07.577413 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.577382 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:07.873865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:07.873771 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:08.196318 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:08.196289 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2"] Oct 30 02:19:08.197312 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:19:08.197274 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7417f30_4ad3_47c8_835d_b0511309cac5.slice/crio-589c2f1209e91a11e5988cd1182349e7a679572325d50836927e741108ed17a3 WatchSource:0}: Error finding container 589c2f1209e91a11e5988cd1182349e7a679572325d50836927e741108ed17a3: Status 404 returned error can't find the container with id 589c2f1209e91a11e5988cd1182349e7a679572325d50836927e741108ed17a3 Oct 30 02:19:09.053561 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:09.053509 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" event={"ID":"e7417f30-4ad3-47c8-835d-b0511309cac5","Type":"ContainerStarted","Data":"15821988c7a655e7b2a971645068fe0a3e47e9dee2dc2af02bd485ff0531f754"} Oct 30 02:19:09.053812 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:09.053581 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" event={"ID":"e7417f30-4ad3-47c8-835d-b0511309cac5","Type":"ContainerStarted","Data":"589c2f1209e91a11e5988cd1182349e7a679572325d50836927e741108ed17a3"} Oct 30 02:19:09.053812 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:09.053675 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:09.100650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:09.100581 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" podStartSLOduration=2.6916807780000003 podStartE2EDuration="3.100565175s" totalImagesPullingTime="408.884397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:09.099153282 +0000 UTC m=+3027.602085503" watchObservedRunningTime="2026-10-30 02:19:09.100565175 +0000 UTC m=+3027.603497395" Oct 30 02:19:15.068398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:15.068363 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:19:17.631918 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:19:17.631875 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:19:20.348942 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:20.348900 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2"] Oct 30 02:19:25.349645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.349583 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" podUID="f8320748-e22f-473d-a50e-e3f877e385b1" containerName="payload-processing" containerID="cri-o://1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83" gracePeriod=25 Oct 30 02:19:25.675674 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.675642 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:19:25.720365 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.720317 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-plugins-config-volume\") pod \"f8320748-e22f-473d-a50e-e3f877e385b1\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " Oct 30 02:19:25.720615 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.720388 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-kube-api-access-c4scm\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-kube-api-access-c4scm\") pod \"f8320748-e22f-473d-a50e-e3f877e385b1\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " Oct 30 02:19:25.720615 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.720491 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-routing-overlay-volume\") pod \"f8320748-e22f-473d-a50e-e3f877e385b1\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " Oct 30 02:19:25.720812 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.720789 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-provider-credentials\") pod \"f8320748-e22f-473d-a50e-e3f877e385b1\" (UID: \"f8320748-e22f-473d-a50e-e3f877e385b1\") " Oct 30 02:19:25.720915 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.720885 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-plugins-config-volume" pod "f8320748-e22f-473d-a50e-e3f877e385b1" (UID: "f8320748-e22f-473d-a50e-e3f877e385b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:25.721189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.721160 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-routing-overlay-volume" pod "f8320748-e22f-473d-a50e-e3f877e385b1" (UID: "f8320748-e22f-473d-a50e-e3f877e385b1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:25.721633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.721598 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:19:25.721750 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.721636 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8320748-e22f-473d-a50e-e3f877e385b1-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:19:25.724436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.724408 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-kube-api-access-c4scm" pod "f8320748-e22f-473d-a50e-e3f877e385b1" (UID: "f8320748-e22f-473d-a50e-e3f877e385b1"). InnerVolumeSpecName "kube-api-access-c4scm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:25.724544 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.724447 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-provider-credentials" pod "f8320748-e22f-473d-a50e-e3f877e385b1" (UID: "f8320748-e22f-473d-a50e-e3f877e385b1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:25.823539 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.823499 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:19:25.823745 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:25.823552 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4scm\" (UniqueName: \"kubernetes.io/projected/f8320748-e22f-473d-a50e-e3f877e385b1-kube-api-access-c4scm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:19:26.225622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:26.225580 5035 generic.go:356] "Generic (PLEG): container finished" podID="f8320748-e22f-473d-a50e-e3f877e385b1" containerID="1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83" exitCode=0 Oct 30 02:19:26.225822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:26.225669 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" Oct 30 02:19:26.225822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:26.225667 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" event={"ID":"f8320748-e22f-473d-a50e-e3f877e385b1","Type":"ContainerDied","Data":"1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83"} Oct 30 02:19:26.225923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:26.225819 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2" event={"ID":"f8320748-e22f-473d-a50e-e3f877e385b1","Type":"ContainerDied","Data":"744160006a0da82f1c16e5ede5c54672b836af4db358f127bafd31b523533c90"} Oct 30 02:19:26.225923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:26.225844 5035 scope.go:122] "RemoveContainer" containerID="1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83" Oct 30 02:19:26.241457 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:26.241427 5035 scope.go:122] "RemoveContainer" containerID="1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83" Oct 30 02:19:26.244674 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:19:26.244610 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83\": container with ID starting with 1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83 not found: ID does not exist" containerID="1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83" Oct 30 02:19:26.244831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:26.244681 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83"} err="failed to get container status \"1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83\": rpc error: code = NotFound desc = could not find container \"1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83\": container with ID starting with 1db3e6f17a0bc9d56e17d4bee82149007fb87bc8c9b565e671c77a1793adcd83 not found: ID does not exist" Oct 30 02:19:26.283999 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:26.283963 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2"] Oct 30 02:19:26.297405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:26.297369 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-c6454bjf2"] Oct 30 02:19:27.645212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:27.645174 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8320748-e22f-473d-a50e-e3f877e385b1" path="/var/lib/kubelet/pods/f8320748-e22f-473d-a50e-e3f877e385b1/volumes" Oct 30 02:19:47.632370 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:19:47.632329 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:19:54.967038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:54.967000 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx"] Oct 30 02:19:59.967441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:19:59.967387 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" podUID="094e006f-b549-4050-a3b0-5b19d663bd4d" containerName="payload-processing" containerID="cri-o://ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9" gracePeriod=25 Oct 30 02:20:00.257389 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.257356 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:20:00.372155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.372113 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-plugins-config-volume\") pod \"094e006f-b549-4050-a3b0-5b19d663bd4d\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " Oct 30 02:20:00.372383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.372347 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-routing-overlay-volume\") pod \"094e006f-b549-4050-a3b0-5b19d663bd4d\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " Oct 30 02:20:00.372465 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.372399 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-kube-api-access-sqwjj\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-kube-api-access-sqwjj\") pod \"094e006f-b549-4050-a3b0-5b19d663bd4d\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " Oct 30 02:20:00.372614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.372582 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-plugins-config-volume" pod "094e006f-b549-4050-a3b0-5b19d663bd4d" (UID: "094e006f-b549-4050-a3b0-5b19d663bd4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:00.372722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.372593 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-provider-credentials\") pod \"094e006f-b549-4050-a3b0-5b19d663bd4d\" (UID: \"094e006f-b549-4050-a3b0-5b19d663bd4d\") " Oct 30 02:20:00.372901 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.372877 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-routing-overlay-volume" pod "094e006f-b549-4050-a3b0-5b19d663bd4d" (UID: "094e006f-b549-4050-a3b0-5b19d663bd4d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:00.373355 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.373331 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.373355 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.373355 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e006f-b549-4050-a3b0-5b19d663bd4d-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.375106 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.375082 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-kube-api-access-sqwjj" pod "094e006f-b549-4050-a3b0-5b19d663bd4d" (UID: "094e006f-b549-4050-a3b0-5b19d663bd4d"). InnerVolumeSpecName "kube-api-access-sqwjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:00.375407 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.375391 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-provider-credentials" pod "094e006f-b549-4050-a3b0-5b19d663bd4d" (UID: "094e006f-b549-4050-a3b0-5b19d663bd4d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:00.475041 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.474943 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.475041 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.474978 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqwjj\" (UniqueName: \"kubernetes.io/projected/094e006f-b549-4050-a3b0-5b19d663bd4d-kube-api-access-sqwjj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.595603 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.595568 5035 generic.go:356] "Generic (PLEG): container finished" podID="094e006f-b549-4050-a3b0-5b19d663bd4d" containerID="ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9" exitCode=0 Oct 30 02:20:00.595803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.595622 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" Oct 30 02:20:00.595803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.595654 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" event={"ID":"094e006f-b549-4050-a3b0-5b19d663bd4d","Type":"ContainerDied","Data":"ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9"} Oct 30 02:20:00.595803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.595717 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx" event={"ID":"094e006f-b549-4050-a3b0-5b19d663bd4d","Type":"ContainerDied","Data":"46a9d4140fd3450d57ed2a0566113a7bd5b25139512fe3f3a478e4592523760a"} Oct 30 02:20:00.595803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.595747 5035 scope.go:122] "RemoveContainer" containerID="ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9" Oct 30 02:20:00.607995 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.607970 5035 scope.go:122] "RemoveContainer" containerID="ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9" Oct 30 02:20:00.608316 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:00.608291 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9\": container with ID starting with ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9 not found: ID does not exist" containerID="ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9" Oct 30 02:20:00.608401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.608329 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9"} err="failed to get container status \"ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9\": rpc error: code = NotFound desc = could not find container \"ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9\": container with ID starting with ba647774102db3fdb469844eb7ceaa1fe042bac64dff7c898d8b4e8d907ec1c9 not found: ID does not exist" Oct 30 02:20:00.666632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.666595 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx"] Oct 30 02:20:00.679630 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:00.679587 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-6f66bc6tx"] Oct 30 02:20:01.647429 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.647392 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094e006f-b549-4050-a3b0-5b19d663bd4d" path="/var/lib/kubelet/pods/094e006f-b549-4050-a3b0-5b19d663bd4d/volumes" Oct 30 02:20:01.867659 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.867606 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj"] Oct 30 02:20:01.868436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.868415 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f8320748-e22f-473d-a50e-e3f877e385b1" containerName="payload-processing" Oct 30 02:20:01.868436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.868434 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8320748-e22f-473d-a50e-e3f877e385b1" containerName="payload-processing" Oct 30 02:20:01.868652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.868471 5035 cpu_manager.go:414] "containerMap: removing container" podUID="094e006f-b549-4050-a3b0-5b19d663bd4d" containerName="payload-processing" Oct 30 02:20:01.868652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.868478 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e006f-b549-4050-a3b0-5b19d663bd4d" containerName="payload-processing" Oct 30 02:20:01.868652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.868617 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="094e006f-b549-4050-a3b0-5b19d663bd4d" containerName="payload-processing" Oct 30 02:20:01.868652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.868634 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8320748-e22f-473d-a50e-e3f877e385b1" containerName="payload-processing" Oct 30 02:20:01.873888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.873862 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:01.892396 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.892360 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj"] Oct 30 02:20:01.900441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.900347 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2c4b44-dockercfg-kn2wp\"" Oct 30 02:20:01.900441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:01.900343 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2c4b44\"" Oct 30 02:20:02.015562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.015498 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfz8\" (UniqueName: \"kubernetes.io/projected/708b87c9-902e-4882-9b48-1ad131b19385-kube-api-access-rvfz8\") pod \"payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj\" (UID: \"708b87c9-902e-4882-9b48-1ad131b19385\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:02.016078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.016056 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708b87c9-902e-4882-9b48-1ad131b19385-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj\" (UID: \"708b87c9-902e-4882-9b48-1ad131b19385\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:02.031207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.031174 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4"] Oct 30 02:20:02.038300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.038252 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:02.049184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.049138 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4"] Oct 30 02:20:02.117998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.117957 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708b87c9-902e-4882-9b48-1ad131b19385-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj\" (UID: \"708b87c9-902e-4882-9b48-1ad131b19385\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:02.117998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.118001 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a90a801-f3b4-439b-b4b5-2a5ece592228-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4\" (UID: \"5a90a801-f3b4-439b-b4b5-2a5ece592228\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:02.118239 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.118179 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfz8\" (UniqueName: \"kubernetes.io/projected/708b87c9-902e-4882-9b48-1ad131b19385-kube-api-access-rvfz8\") pod \"payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj\" (UID: \"708b87c9-902e-4882-9b48-1ad131b19385\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:02.118279 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.118263 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9xkv\" (UniqueName: \"kubernetes.io/projected/5a90a801-f3b4-439b-b4b5-2a5ece592228-kube-api-access-q9xkv\") pod \"payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4\" (UID: \"5a90a801-f3b4-439b-b4b5-2a5ece592228\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:02.118830 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.118809 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708b87c9-902e-4882-9b48-1ad131b19385-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj\" (UID: \"708b87c9-902e-4882-9b48-1ad131b19385\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:02.144322 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.144290 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfz8\" (UniqueName: \"kubernetes.io/projected/708b87c9-902e-4882-9b48-1ad131b19385-kube-api-access-rvfz8\") pod \"payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj\" (UID: \"708b87c9-902e-4882-9b48-1ad131b19385\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:02.193504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.193419 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:02.219494 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.219456 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9xkv\" (UniqueName: \"kubernetes.io/projected/5a90a801-f3b4-439b-b4b5-2a5ece592228-kube-api-access-q9xkv\") pod \"payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4\" (UID: \"5a90a801-f3b4-439b-b4b5-2a5ece592228\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:02.219871 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.219606 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a90a801-f3b4-439b-b4b5-2a5ece592228-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4\" (UID: \"5a90a801-f3b4-439b-b4b5-2a5ece592228\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:02.220594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.220522 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a90a801-f3b4-439b-b4b5-2a5ece592228-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4\" (UID: \"5a90a801-f3b4-439b-b4b5-2a5ece592228\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:02.245387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.245352 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9xkv\" (UniqueName: \"kubernetes.io/projected/5a90a801-f3b4-439b-b4b5-2a5ece592228-kube-api-access-q9xkv\") pod \"payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4\" (UID: \"5a90a801-f3b4-439b-b4b5-2a5ece592228\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:02.382989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.382949 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:02.514684 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.514647 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj"] Oct 30 02:20:02.515477 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:20:02.515442 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708b87c9_902e_4882_9b48_1ad131b19385.slice/crio-e2b2e6e9b6580b036d08057f803a1fd2d15288dc5d6026c03f1ca2bf528e17aa WatchSource:0}: Error finding container e2b2e6e9b6580b036d08057f803a1fd2d15288dc5d6026c03f1ca2bf528e17aa: Status 404 returned error can't find the container with id e2b2e6e9b6580b036d08057f803a1fd2d15288dc5d6026c03f1ca2bf528e17aa Oct 30 02:20:02.618961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.618918 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" event={"ID":"708b87c9-902e-4882-9b48-1ad131b19385","Type":"ContainerStarted","Data":"e2b2e6e9b6580b036d08057f803a1fd2d15288dc5d6026c03f1ca2bf528e17aa"} Oct 30 02:20:02.699265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:02.699230 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4"] Oct 30 02:20:02.700765 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:20:02.700733 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a90a801_f3b4_439b_b4b5_2a5ece592228.slice/crio-348e56a1ab206e30226262bf14315fefd3ced32eb9ffeef0ea7b6f88dae75a43 WatchSource:0}: Error finding container 348e56a1ab206e30226262bf14315fefd3ced32eb9ffeef0ea7b6f88dae75a43: Status 404 returned error can't find the container with id 348e56a1ab206e30226262bf14315fefd3ced32eb9ffeef0ea7b6f88dae75a43 Oct 30 02:20:03.628413 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:03.628365 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" event={"ID":"708b87c9-902e-4882-9b48-1ad131b19385","Type":"ContainerStarted","Data":"31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb"} Oct 30 02:20:03.628671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:03.628651 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:03.630286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:03.630250 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" event={"ID":"5a90a801-f3b4-439b-b4b5-2a5ece592228","Type":"ContainerStarted","Data":"749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24"} Oct 30 02:20:03.630431 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:03.630303 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" event={"ID":"5a90a801-f3b4-439b-b4b5-2a5ece592228","Type":"ContainerStarted","Data":"348e56a1ab206e30226262bf14315fefd3ced32eb9ffeef0ea7b6f88dae75a43"} Oct 30 02:20:03.630431 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:03.630360 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:03.677214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:03.677151 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" podStartSLOduration=2.677134711 podStartE2EDuration="2.677134711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:03.673093345 +0000 UTC m=+3082.176025577" watchObservedRunningTime="2026-10-30 02:20:03.677134711 +0000 UTC m=+3082.180067154" Oct 30 02:20:03.721399 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:03.721338 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" podStartSLOduration=2.721320307 podStartE2EDuration="2.721320307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:03.71758028 +0000 UTC m=+3082.220512503" watchObservedRunningTime="2026-10-30 02:20:03.721320307 +0000 UTC m=+3082.224252568" Oct 30 02:20:14.650942 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:14.650892 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:20:14.651434 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:14.651026 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:20:17.631023 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:17.630972 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:20:24.032729 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.032632 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh"] Oct 30 02:20:24.037741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.037707 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.070021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.069687 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"openshift-service-ca.crt\"" Oct 30 02:20:24.071122 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.071093 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"kube-root-ca.crt\"" Oct 30 02:20:24.071637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.071615 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"routing-overlay\"" Oct 30 02:20:24.072104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.072086 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"agc-cred-2a752186c169\"" Oct 30 02:20:24.072397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.072379 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"payload-processing-external-model-e2e--b7f31742-dockercfg-cjhl5\"" Oct 30 02:20:24.072577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.072562 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6b8bfa\"" Oct 30 02:20:24.103732 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.075409 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh"] Oct 30 02:20:24.160563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.158008 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fn8\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-kube-api-access-j7fn8\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.160563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.158120 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.160563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.158180 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.160563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.158229 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.259881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.259844 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fn8\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-kube-api-access-j7fn8\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.260104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.259913 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.260104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.259968 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.260104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.260008 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.260741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.260707 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.260858 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.260738 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.262901 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.262871 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.289775 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.289684 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fn8\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-kube-api-access-j7fn8\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.369664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.369630 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.687385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.687356 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh"] Oct 30 02:20:24.688265 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:20:24.688234 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2651547c_1576_4e03_95e1_87eb690f89d8.slice/crio-749a4ab702a0e2ada385725952fc57775c3a915ef54c17aae4371461836f0fa6 WatchSource:0}: Error finding container 749a4ab702a0e2ada385725952fc57775c3a915ef54c17aae4371461836f0fa6: Status 404 returned error can't find the container with id 749a4ab702a0e2ada385725952fc57775c3a915ef54c17aae4371461836f0fa6 Oct 30 02:20:24.843027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.842982 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" event={"ID":"2651547c-1576-4e03-95e1-87eb690f89d8","Type":"ContainerStarted","Data":"4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974"} Oct 30 02:20:24.843228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.843053 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" event={"ID":"2651547c-1576-4e03-95e1-87eb690f89d8","Type":"ContainerStarted","Data":"749a4ab702a0e2ada385725952fc57775c3a915ef54c17aae4371461836f0fa6"} Oct 30 02:20:24.843228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.843081 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:24.898834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:24.898727 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" podStartSLOduration=1.898500326 podStartE2EDuration="1.898500326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:24.893992985 +0000 UTC m=+3103.396925208" watchObservedRunningTime="2026-10-30 02:20:24.898500326 +0000 UTC m=+3103.401432546" Oct 30 02:20:25.925167 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:25.925124 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:25.925167 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:25.925162 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh: secret "agc-cred-2a752186c169" not found Oct 30 02:20:25.925825 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:25.925230 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials podName:2651547c-1576-4e03-95e1-87eb690f89d8 nodeName:}" failed. No retries permitted until 2026-10-30 02:20:26.425210912 +0000 UTC m=+3104.928143110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" (UID: "2651547c-1576-4e03-95e1-87eb690f89d8") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:26.443552 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:26.437500 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:26.443552 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:26.437556 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh: secret "agc-cred-2a752186c169" not found Oct 30 02:20:26.443552 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:26.437631 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials podName:2651547c-1576-4e03-95e1-87eb690f89d8 nodeName:}" failed. No retries permitted until 2026-10-30 02:20:27.437609165 +0000 UTC m=+3105.940541381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" (UID: "2651547c-1576-4e03-95e1-87eb690f89d8") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:27.332071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:27.332024 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh"] Oct 30 02:20:27.459573 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:27.459504 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:27.459573 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:27.459572 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh: secret "agc-cred-2a752186c169" not found Oct 30 02:20:27.459790 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:27.459634 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials podName:2651547c-1576-4e03-95e1-87eb690f89d8 nodeName:}" failed. No retries permitted until 2026-10-30 02:20:29.459618391 +0000 UTC m=+3107.962550592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" (UID: "2651547c-1576-4e03-95e1-87eb690f89d8") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:29.497885 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:29.497838 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6b8bfa/agc-cred-2a752186c169: secret "agc-cred-2a752186c169" not found Oct 30 02:20:29.497885 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:29.497875 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh: secret "agc-cred-2a752186c169" not found Oct 30 02:20:29.498409 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:29.497954 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials podName:2651547c-1576-4e03-95e1-87eb690f89d8 nodeName:}" failed. No retries permitted until 2026-10-30 02:20:33.497930404 +0000 UTC m=+3112.000862622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" (UID: "2651547c-1576-4e03-95e1-87eb690f89d8") : secret "agc-cred-2a752186c169" not found Oct 30 02:20:30.912004 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.911962 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l"] Oct 30 02:20:30.917118 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.917092 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:30.934337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.934291 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"agc-cred-80a3dc502ca9\"" Oct 30 02:20:30.934503 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.934346 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"agc-cred-f203f6413ee7\"" Oct 30 02:20:30.934584 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.934524 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"payload-processing-external-model-e2e--1bd5e061-dockercfg-jz6pw\"" Oct 30 02:20:30.948209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.948176 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"openshift-service-ca.crt\"" Oct 30 02:20:30.948209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.948177 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a0fc73\"" Oct 30 02:20:30.948209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.948194 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"routing-overlay\"" Oct 30 02:20:30.948568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.948254 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"kube-root-ca.crt\"" Oct 30 02:20:30.954365 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:30.954327 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l"] Oct 30 02:20:31.075419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.075365 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.075673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.075434 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.075673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.075628 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.075777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.075742 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-kube-api-access-hhwpg\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.177765 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.177664 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.177765 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.177754 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.177996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.177822 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.177996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.177907 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-kube-api-access-hhwpg\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.178455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.178435 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.178643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.178621 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.180585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.180560 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.203135 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.203099 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-kube-api-access-hhwpg\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.234383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.234348 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.551209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.551175 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l"] Oct 30 02:20:31.552051 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:20:31.552020 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f69e4d2_3ee7_42f1_912b_8f259e303363.slice/crio-725e2e0e1f58b3d6c19809e3dc5a307504f3f585a18753533355969e275e7f5f WatchSource:0}: Error finding container 725e2e0e1f58b3d6c19809e3dc5a307504f3f585a18753533355969e275e7f5f: Status 404 returned error can't find the container with id 725e2e0e1f58b3d6c19809e3dc5a307504f3f585a18753533355969e275e7f5f Oct 30 02:20:31.899795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.899681 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" event={"ID":"1f69e4d2-3ee7-42f1-912b-8f259e303363","Type":"ContainerStarted","Data":"29dc9bbac2d1d100ebe1d2994c1ab644f691ec83fba9471c9acd678426773bef"} Oct 30 02:20:31.899795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.899737 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" event={"ID":"1f69e4d2-3ee7-42f1-912b-8f259e303363","Type":"ContainerStarted","Data":"725e2e0e1f58b3d6c19809e3dc5a307504f3f585a18753533355969e275e7f5f"} Oct 30 02:20:31.899795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.899788 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:31.946487 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:31.946425 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" podStartSLOduration=1.9464099209999999 podStartE2EDuration="1.946409921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:31.945104971 +0000 UTC m=+3110.448037192" watchObservedRunningTime="2026-10-30 02:20:31.946409921 +0000 UTC m=+3110.449342166" Oct 30 02:20:32.333275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.333227 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" podUID="2651547c-1576-4e03-95e1-87eb690f89d8" containerName="payload-processing" containerID="cri-o://4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974" gracePeriod=25 Oct 30 02:20:32.618712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.618682 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:32.700444 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.700401 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials\") pod \"2651547c-1576-4e03-95e1-87eb690f89d8\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " Oct 30 02:20:32.700662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.700596 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-kube-api-access-j7fn8\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-kube-api-access-j7fn8\") pod \"2651547c-1576-4e03-95e1-87eb690f89d8\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " Oct 30 02:20:32.700767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.700747 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-routing-overlay-volume\") pod \"2651547c-1576-4e03-95e1-87eb690f89d8\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " Oct 30 02:20:32.700831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.700786 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-plugins-config-volume\") pod \"2651547c-1576-4e03-95e1-87eb690f89d8\" (UID: \"2651547c-1576-4e03-95e1-87eb690f89d8\") " Oct 30 02:20:32.701275 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.701240 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-routing-overlay-volume" pod "2651547c-1576-4e03-95e1-87eb690f89d8" (UID: "2651547c-1576-4e03-95e1-87eb690f89d8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:32.701509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.701410 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:20:32.701681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.701657 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-plugins-config-volume" pod "2651547c-1576-4e03-95e1-87eb690f89d8" (UID: "2651547c-1576-4e03-95e1-87eb690f89d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:32.703771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.703731 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-kube-api-access-j7fn8" pod "2651547c-1576-4e03-95e1-87eb690f89d8" (UID: "2651547c-1576-4e03-95e1-87eb690f89d8"). InnerVolumeSpecName "kube-api-access-j7fn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:32.703916 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.703894 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials" pod "2651547c-1576-4e03-95e1-87eb690f89d8" (UID: "2651547c-1576-4e03-95e1-87eb690f89d8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:32.802860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.802823 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2651547c-1576-4e03-95e1-87eb690f89d8-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:20:32.802860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.802855 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:20:32.802860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.802866 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7fn8\" (UniqueName: \"kubernetes.io/projected/2651547c-1576-4e03-95e1-87eb690f89d8-kube-api-access-j7fn8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:20:32.909900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.909784 5035 generic.go:356] "Generic (PLEG): container finished" podID="2651547c-1576-4e03-95e1-87eb690f89d8" containerID="4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974" exitCode=0 Oct 30 02:20:32.909900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.909844 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" Oct 30 02:20:32.909900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.909869 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" event={"ID":"2651547c-1576-4e03-95e1-87eb690f89d8","Type":"ContainerDied","Data":"4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974"} Oct 30 02:20:32.910222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.909943 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh" event={"ID":"2651547c-1576-4e03-95e1-87eb690f89d8","Type":"ContainerDied","Data":"749a4ab702a0e2ada385725952fc57775c3a915ef54c17aae4371461836f0fa6"} Oct 30 02:20:32.910222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.909976 5035 scope.go:122] "RemoveContainer" containerID="4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974" Oct 30 02:20:32.922378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.922355 5035 scope.go:122] "RemoveContainer" containerID="4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974" Oct 30 02:20:32.922729 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:32.922705 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974\": container with ID starting with 4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974 not found: ID does not exist" containerID="4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974" Oct 30 02:20:32.922832 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.922736 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974"} err="failed to get container status \"4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974\": rpc error: code = NotFound desc = could not find container \"4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974\": container with ID starting with 4d5d48b3d048ab87e5d37baf973209bcc2e390a866f08f75c027c966bf783974 not found: ID does not exist" Oct 30 02:20:32.968507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.968466 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh"] Oct 30 02:20:32.975777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:32.975743 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-646bn5dsh"] Oct 30 02:20:33.646219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:33.646184 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2651547c-1576-4e03-95e1-87eb690f89d8" path="/var/lib/kubelet/pods/2651547c-1576-4e03-95e1-87eb690f89d8/volumes" Oct 30 02:20:42.342601 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.342561 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb"] Oct 30 02:20:42.345047 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.343344 5035 cpu_manager.go:414] "containerMap: removing container" podUID="2651547c-1576-4e03-95e1-87eb690f89d8" containerName="payload-processing" Oct 30 02:20:42.345047 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.343357 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2651547c-1576-4e03-95e1-87eb690f89d8" containerName="payload-processing" Oct 30 02:20:42.345047 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.343504 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="2651547c-1576-4e03-95e1-87eb690f89d8" containerName="payload-processing" Oct 30 02:20:42.347282 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.347260 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.370178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.370133 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb"] Oct 30 02:20:42.377954 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.377918 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2c4b44-serving-cert\"" Oct 30 02:20:42.448341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.448288 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-metrics-certs\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.448563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.448371 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20aac51-0f1a-4f75-81d3-33f010361b2e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.448563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.448501 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvf6f\" (UniqueName: \"kubernetes.io/projected/d20aac51-0f1a-4f75-81d3-33f010361b2e-kube-api-access-hvf6f\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.448650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.448566 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.564566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.562215 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-metrics-certs\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.564566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.562767 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20aac51-0f1a-4f75-81d3-33f010361b2e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.564566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.562923 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf6f\" (UniqueName: \"kubernetes.io/projected/d20aac51-0f1a-4f75-81d3-33f010361b2e-kube-api-access-hvf6f\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.564566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.563000 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.564566 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:42.563218 5035 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-serving-cert: secret "maas-api-e2e-extmod-b-2c4b44-serving-cert" not found Oct 30 02:20:42.564566 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:42.563289 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls podName:d20aac51-0f1a-4f75-81d3-33f010361b2e nodeName:}" failed. No retries permitted until 2026-10-30 02:20:43.063265485 +0000 UTC m=+3121.566197700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls") pod "maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" (UID: "d20aac51-0f1a-4f75-81d3-33f010361b2e") : secret "maas-api-e2e-extmod-b-2c4b44-serving-cert" not found Oct 30 02:20:42.564566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.564462 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20aac51-0f1a-4f75-81d3-33f010361b2e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.568237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.568206 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-metrics-certs\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.589951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.589912 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvf6f\" (UniqueName: \"kubernetes.io/projected/d20aac51-0f1a-4f75-81d3-33f010361b2e-kube-api-access-hvf6f\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:42.914260 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:42.914227 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:20:43.072370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:43.072315 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:43.075008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:43.074974 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:43.263942 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:43.263890 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:43.592067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:43.592032 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb"] Oct 30 02:20:43.592789 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:20:43.592758 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20aac51_0f1a_4f75_81d3_33f010361b2e.slice/crio-ba8b8b1ab113e7a09a9f10399e6f39b47a2a25e812e35775724903c3cb58cd76 WatchSource:0}: Error finding container ba8b8b1ab113e7a09a9f10399e6f39b47a2a25e812e35775724903c3cb58cd76: Status 404 returned error can't find the container with id ba8b8b1ab113e7a09a9f10399e6f39b47a2a25e812e35775724903c3cb58cd76 Oct 30 02:20:43.594682 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:43.594661 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:20:44.029760 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:44.029720 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" event={"ID":"d20aac51-0f1a-4f75-81d3-33f010361b2e","Type":"ContainerStarted","Data":"996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8"} Oct 30 02:20:44.029965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:44.029779 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" event={"ID":"d20aac51-0f1a-4f75-81d3-33f010361b2e","Type":"ContainerStarted","Data":"ba8b8b1ab113e7a09a9f10399e6f39b47a2a25e812e35775724903c3cb58cd76"} Oct 30 02:20:44.029965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:44.029839 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:44.137721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:44.137601 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" podStartSLOduration=1.815100901 podStartE2EDuration="2.137584412s" totalImagesPullingTime="322.483511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:44.134756593 +0000 UTC m=+3122.637688813" watchObservedRunningTime="2026-10-30 02:20:44.137584412 +0000 UTC m=+3122.640516632" Oct 30 02:20:45.899146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:45.899106 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7"] Oct 30 02:20:45.904489 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:45.904455 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:45.923889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:45.923782 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5dmpm\"" Oct 30 02:20:45.928849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:45.928813 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7"] Oct 30 02:20:46.023982 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.023922 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.024144 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.024031 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.024144 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.024060 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.024249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.024165 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.024249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.024223 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.024478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.024456 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.024593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.024577 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.024634 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.024606 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqcwf\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-kube-api-access-sqcwf\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.024670 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.024630 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126145 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126194 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqcwf\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-kube-api-access-sqcwf\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126220 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126253 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126272 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126292 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126343 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126424 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126647 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126820 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.126906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126857 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.127161 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.126970 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.127161 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.127054 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.127504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.127481 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.129048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.129022 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.129163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.129082 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.151336 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.151260 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.151610 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.151583 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqcwf\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-kube-api-access-sqcwf\") pod \"maas-default-gateway-openshift-default-75d7c785f-2wbd7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.220906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.220863 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:46.553419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.553382 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7"] Oct 30 02:20:46.554583 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:20:46.554552 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29aa42b6_d6e4_417e_8e13_550d0e7020c7.slice/crio-753f6e06742659c858f05529b550bb1008e1d38d9552c644478fc6a463c63931 WatchSource:0}: Error finding container 753f6e06742659c858f05529b550bb1008e1d38d9552c644478fc6a463c63931: Status 404 returned error can't find the container with id 753f6e06742659c858f05529b550bb1008e1d38d9552c644478fc6a463c63931 Oct 30 02:20:46.558188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.558152 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:20:46.558286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.558227 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:20:46.558286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:46.558257 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:20:47.057911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:47.057874 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" event={"ID":"29aa42b6-d6e4-417e-8e13-550d0e7020c7","Type":"ContainerStarted","Data":"c09fa3242ce1a7a69f45561aff60ea94ad13b2ffa0adfe854e276ecea7554704"} Oct 30 02:20:47.058382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:47.057951 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" event={"ID":"29aa42b6-d6e4-417e-8e13-550d0e7020c7","Type":"ContainerStarted","Data":"753f6e06742659c858f05529b550bb1008e1d38d9552c644478fc6a463c63931"} Oct 30 02:20:47.124276 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:47.124199 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podStartSLOduration=2.12417514 podStartE2EDuration="2.12417514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:47.120497207 +0000 UTC m=+3125.623429427" watchObservedRunningTime="2026-10-30 02:20:47.12417514 +0000 UTC m=+3125.627107362" Oct 30 02:20:47.221787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:47.221743 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:47.224146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:47.224117 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 02:20:47.224305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:47.224170 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 02:20:47.631464 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:20:47.631426 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:20:48.222201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:48.222153 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 02:20:48.222648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:48.222209 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 02:20:49.222001 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:49.221961 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 02:20:49.222337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:49.222024 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 02:20:50.044019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:50.043984 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:20:50.221853 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:50.221816 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 02:20:50.222043 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:50.221879 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 02:20:51.222017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:51.221973 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 02:20:51.222472 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:51.222034 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 02:20:52.222224 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:52.222188 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 02:20:52.222750 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:52.222248 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 02:20:53.221957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:53.221911 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 02:20:53.222202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:53.221979 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 02:20:54.221467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:54.221421 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 30 02:20:54.221921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:54.221518 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 30 02:20:55.350617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:55.350584 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:55.351101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:55.351076 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:20:55.352105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:20:55.351949 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:21:00.999349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:00.999302 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2"] Oct 30 02:21:01.368453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.368359 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2"] Oct 30 02:21:01.368793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.368501 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.466315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.466276 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d77144-491e-48c4-80c2-530ab45394cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.466519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.466336 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.466597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.466512 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d77144-491e-48c4-80c2-530ab45394cd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.466654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.466638 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.466692 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.466668 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.466755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.466737 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2sxz\" (UniqueName: \"kubernetes.io/projected/27d77144-491e-48c4-80c2-530ab45394cd-kube-api-access-h2sxz\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.466801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.466790 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d77144-491e-48c4-80c2-530ab45394cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.466839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.466819 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.466931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.466919 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.568667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.568624 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d77144-491e-48c4-80c2-530ab45394cd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.568890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.568727 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.568890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.568760 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.568890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.568801 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2sxz\" (UniqueName: \"kubernetes.io/projected/27d77144-491e-48c4-80c2-530ab45394cd-kube-api-access-h2sxz\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.568890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.568849 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d77144-491e-48c4-80c2-530ab45394cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.568890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.568884 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.569172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.568933 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.569172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.569038 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d77144-491e-48c4-80c2-530ab45394cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.569172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.569112 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.569363 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.569323 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.569432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.569365 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.569432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.569394 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.569575 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.569493 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.569952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.569927 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d77144-491e-48c4-80c2-530ab45394cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.571610 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.571577 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d77144-491e-48c4-80c2-530ab45394cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.571732 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.571663 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d77144-491e-48c4-80c2-530ab45394cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.595798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.595766 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d77144-491e-48c4-80c2-530ab45394cd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.596086 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.596056 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2sxz\" (UniqueName: \"kubernetes.io/projected/27d77144-491e-48c4-80c2-530ab45394cd-kube-api-access-h2sxz\") pod \"maas-default-gateway-openshift-default-75d7c785f-rt6p2\" (UID: \"27d77144-491e-48c4-80c2-530ab45394cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:01.701875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:01.701833 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:02.047311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:02.047272 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2"] Oct 30 02:21:02.047721 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:21:02.047332 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d77144_491e_48c4_80c2_530ab45394cd.slice/crio-0ee690950e2225f0dbef5289c8ccc6aa233808b22c38d6b121225ba668ca052a WatchSource:0}: Error finding container 0ee690950e2225f0dbef5289c8ccc6aa233808b22c38d6b121225ba668ca052a: Status 404 returned error can't find the container with id 0ee690950e2225f0dbef5289c8ccc6aa233808b22c38d6b121225ba668ca052a Oct 30 02:21:02.051184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:02.051138 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:21:02.051297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:02.051275 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:21:02.051347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:02.051331 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:21:02.353650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:02.353544 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" event={"ID":"27d77144-491e-48c4-80c2-530ab45394cd","Type":"ContainerStarted","Data":"1fcf88ccb88c50cb6197bb73c9e428c5a8418146539e60275d887dfecc4c5cbe"} Oct 30 02:21:02.353650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:02.353614 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" event={"ID":"27d77144-491e-48c4-80c2-530ab45394cd","Type":"ContainerStarted","Data":"0ee690950e2225f0dbef5289c8ccc6aa233808b22c38d6b121225ba668ca052a"} Oct 30 02:21:02.412064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:02.411991 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" podStartSLOduration=2.411968312 podStartE2EDuration="2.411968312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:02.408743353 +0000 UTC m=+3140.911675573" watchObservedRunningTime="2026-10-30 02:21:02.411968312 +0000 UTC m=+3140.914900532" Oct 30 02:21:02.702863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:02.702827 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:03.704649 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:03.704590 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rt6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:21:03.705191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:03.704664 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" podUID="27d77144-491e-48c4-80c2-530ab45394cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 02:21:04.702917 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:04.702874 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rt6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:21:04.703118 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:04.702937 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" podUID="27d77144-491e-48c4-80c2-530ab45394cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 02:21:05.703238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:05.703197 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rt6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:21:05.703709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:05.703282 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" podUID="27d77144-491e-48c4-80c2-530ab45394cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 02:21:06.702242 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:06.702202 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rt6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:21:06.702457 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:06.702275 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" podUID="27d77144-491e-48c4-80c2-530ab45394cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 02:21:07.702386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:07.702347 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rt6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:21:07.702962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:07.702404 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" podUID="27d77144-491e-48c4-80c2-530ab45394cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 02:21:08.702834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:08.702786 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rt6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:21:08.703391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:08.702866 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" podUID="27d77144-491e-48c4-80c2-530ab45394cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 02:21:09.702368 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:09.702317 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-rt6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:21:09.702641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:09.702388 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" podUID="27d77144-491e-48c4-80c2-530ab45394cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 02:21:10.865445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:10.865409 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:10.866005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:10.865948 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:10.866997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:10.866973 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-rt6p2" Oct 30 02:21:16.023287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.023239 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2"] Oct 30 02:21:16.030448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.030420 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.065237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.065195 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2"] Oct 30 02:21:16.167928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.167208 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.167928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.167312 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.167928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.167341 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.167928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.167422 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwl8h\" (UniqueName: \"kubernetes.io/projected/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-kube-api-access-hwl8h\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.167928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.167459 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.167928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.167496 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.167928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.167563 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.167928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.167607 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.167928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.167748 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.298904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.298766 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.298904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.298859 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.298904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.298889 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.299221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.298965 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwl8h\" (UniqueName: \"kubernetes.io/projected/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-kube-api-access-hwl8h\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.299221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.298996 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.299221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.299028 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.299221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.299073 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.299221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.299111 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.299497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.299242 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.299828 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.299803 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.300698 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.300507 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.301005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.300867 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.334565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.333389 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.338565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.334824 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.338782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.338648 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.350553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.340217 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.362625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.360709 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwl8h\" (UniqueName: \"kubernetes.io/projected/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-kube-api-access-hwl8h\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.381635 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.377103 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dae7370-d6fd-4ad6-bcc3-2bfb55be9904-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-g55b2\" (UID: \"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.389955 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.388776 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:16.822785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.822194 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:21:16.822785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.822303 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:21:16.822785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.822345 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:21:16.834424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:16.833654 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2"] Oct 30 02:21:17.518061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:17.518015 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" event={"ID":"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904","Type":"ContainerStarted","Data":"fd214b40f252cf5ada28c76f429eb812310af8e2a16dd7089bcfe2d8623e3ec7"} Oct 30 02:21:17.518503 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:17.518083 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" event={"ID":"4dae7370-d6fd-4ad6-bcc3-2bfb55be9904","Type":"ContainerStarted","Data":"05a60ad5a5c0a557f0c7f3eb5776d77eeadd07ba48cddf647547750747e967f7"} Oct 30 02:21:17.638893 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:21:17.638839 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:21:18.396553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:18.392740 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:18.642984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:18.642928 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-g55b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 02:21:18.643608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:18.643031 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" podUID="4dae7370-d6fd-4ad6-bcc3-2bfb55be9904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 02:21:19.390156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:19.390114 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-g55b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 02:21:19.390491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:19.390455 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" podUID="4dae7370-d6fd-4ad6-bcc3-2bfb55be9904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 02:21:20.393179 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:20.393130 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-g55b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 02:21:20.393933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:20.393215 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" podUID="4dae7370-d6fd-4ad6-bcc3-2bfb55be9904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 02:21:21.390202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:21.390161 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-g55b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 02:21:21.390977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:21.390499 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" podUID="4dae7370-d6fd-4ad6-bcc3-2bfb55be9904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 02:21:22.391114 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:22.391069 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-g55b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 02:21:22.391845 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:22.391812 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" podUID="4dae7370-d6fd-4ad6-bcc3-2bfb55be9904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 02:21:23.392097 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:23.392037 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-g55b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 02:21:23.392601 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:23.392128 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" podUID="4dae7370-d6fd-4ad6-bcc3-2bfb55be9904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 02:21:24.390600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:24.390491 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-g55b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 02:21:24.390830 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:24.390661 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" podUID="4dae7370-d6fd-4ad6-bcc3-2bfb55be9904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 02:21:26.305823 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:26.305779 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:26.306457 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:26.306252 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:26.307273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:26.307248 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" Oct 30 02:21:26.384736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:21:26.384672 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-g55b2" podStartSLOduration=11.384652175 podStartE2EDuration="11.384652175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:17.605594331 +0000 UTC m=+3156.108526554" watchObservedRunningTime="2026-10-30 02:21:26.384652175 +0000 UTC m=+3164.887584395" Oct 30 02:21:47.636591 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:21:47.636553 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:22:17.632624 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:22:17.632515 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:22:47.631676 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:22:47.631635 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:23:17.871563 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:23:17.866999 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:23:18.907459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:18.907417 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l"] Oct 30 02:23:23.908674 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:23.908569 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" podUID="1f69e4d2-3ee7-42f1-912b-8f259e303363" containerName="payload-processing" containerID="cri-o://29dc9bbac2d1d100ebe1d2994c1ab644f691ec83fba9471c9acd678426773bef" gracePeriod=25 Oct 30 02:23:24.189704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.189659 5035 generic.go:356] "Generic (PLEG): container finished" podID="1f69e4d2-3ee7-42f1-912b-8f259e303363" containerID="29dc9bbac2d1d100ebe1d2994c1ab644f691ec83fba9471c9acd678426773bef" exitCode=0 Oct 30 02:23:24.189918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.189731 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" event={"ID":"1f69e4d2-3ee7-42f1-912b-8f259e303363","Type":"ContainerDied","Data":"29dc9bbac2d1d100ebe1d2994c1ab644f691ec83fba9471c9acd678426773bef"} Oct 30 02:23:24.189918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.189778 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" event={"ID":"1f69e4d2-3ee7-42f1-912b-8f259e303363","Type":"ContainerDied","Data":"725e2e0e1f58b3d6c19809e3dc5a307504f3f585a18753533355969e275e7f5f"} Oct 30 02:23:24.189918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.189795 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="725e2e0e1f58b3d6c19809e3dc5a307504f3f585a18753533355969e275e7f5f" Oct 30 02:23:24.202872 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.202840 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:23:24.294881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.294849 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-plugins-config-volume\") pod \"1f69e4d2-3ee7-42f1-912b-8f259e303363\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " Oct 30 02:23:24.295095 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.294960 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-kube-api-access-hhwpg\") pod \"1f69e4d2-3ee7-42f1-912b-8f259e303363\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " Oct 30 02:23:24.295095 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.295049 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-provider-credentials\") pod \"1f69e4d2-3ee7-42f1-912b-8f259e303363\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " Oct 30 02:23:24.295296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.295279 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-routing-overlay-volume\") pod \"1f69e4d2-3ee7-42f1-912b-8f259e303363\" (UID: \"1f69e4d2-3ee7-42f1-912b-8f259e303363\") " Oct 30 02:23:24.296209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.296165 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-routing-overlay-volume" pod "1f69e4d2-3ee7-42f1-912b-8f259e303363" (UID: "1f69e4d2-3ee7-42f1-912b-8f259e303363"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:23:24.296508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.296490 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-plugins-config-volume" pod "1f69e4d2-3ee7-42f1-912b-8f259e303363" (UID: "1f69e4d2-3ee7-42f1-912b-8f259e303363"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:23:24.299239 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.299208 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-kube-api-access-hhwpg" pod "1f69e4d2-3ee7-42f1-912b-8f259e303363" (UID: "1f69e4d2-3ee7-42f1-912b-8f259e303363"). InnerVolumeSpecName "kube-api-access-hhwpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:23:24.300089 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.300061 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-provider-credentials" pod "1f69e4d2-3ee7-42f1-912b-8f259e303363" (UID: "1f69e4d2-3ee7-42f1-912b-8f259e303363"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:23:24.397469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.397422 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:23:24.397469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.397463 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhwpg\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-kube-api-access-hhwpg\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:23:24.397754 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.397487 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f69e4d2-3ee7-42f1-912b-8f259e303363-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:23:24.397754 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:24.397497 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f69e4d2-3ee7-42f1-912b-8f259e303363-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:23:25.197312 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:25.197274 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l" Oct 30 02:23:25.259881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:25.259837 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l"] Oct 30 02:23:25.272620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:25.272575 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-65fbk725l"] Oct 30 02:23:25.646116 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:25.646080 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f69e4d2-3ee7-42f1-912b-8f259e303363" path="/var/lib/kubelet/pods/1f69e4d2-3ee7-42f1-912b-8f259e303363/volumes" Oct 30 02:23:35.388783 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.388733 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-96b798988-vwbd2"] Oct 30 02:23:35.394721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.394690 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1f69e4d2-3ee7-42f1-912b-8f259e303363" containerName="payload-processing" Oct 30 02:23:35.396260 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.396228 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f69e4d2-3ee7-42f1-912b-8f259e303363" containerName="payload-processing" Oct 30 02:23:35.396894 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.396870 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f69e4d2-3ee7-42f1-912b-8f259e303363" containerName="payload-processing" Oct 30 02:23:35.413864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.413826 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.422432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.422394 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-96b798988-vwbd2"] Oct 30 02:23:35.443728 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.436353 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 02:23:35.443728 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.437079 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1cd01ee51e0b\"" Oct 30 02:23:35.459553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.453902 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-routing-overlay-volume\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.459553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.454000 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-provider-credentials\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.459553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.454155 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-plugins-config-volume\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.459553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.454230 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v58pv\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-kube-api-access-v58pv\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.460950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.460100 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 02:23:35.460950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.460243 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 02:23:35.460950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.460305 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 02:23:35.555605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.555507 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-routing-overlay-volume\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.555808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.555634 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-provider-credentials\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.555808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.555760 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-plugins-config-volume\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.555959 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.555815 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v58pv\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-kube-api-access-v58pv\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.556381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.556348 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-routing-overlay-volume\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.556621 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.556598 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-plugins-config-volume\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.558401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.558374 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-provider-credentials\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.582423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.582387 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v58pv\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-kube-api-access-v58pv\") pod \"payload-processing-external-model-96b798988-vwbd2\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:35.765887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:35.765844 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:36.185632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:36.185598 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-96b798988-vwbd2"] Oct 30 02:23:36.186007 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:23:36.185974 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce2ded4_b790_40da_b518_cfb94db2760d.slice/crio-c9153223047912d8f774d06aabaa0eddc39141eba2ec3a1043649258c54c5fef WatchSource:0}: Error finding container c9153223047912d8f774d06aabaa0eddc39141eba2ec3a1043649258c54c5fef: Status 404 returned error can't find the container with id c9153223047912d8f774d06aabaa0eddc39141eba2ec3a1043649258c54c5fef Oct 30 02:23:36.386454 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:36.386408 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" event={"ID":"dce2ded4-b790-40da-b518-cfb94db2760d","Type":"ContainerStarted","Data":"2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0"} Oct 30 02:23:36.386693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:36.386479 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" event={"ID":"dce2ded4-b790-40da-b518-cfb94db2760d","Type":"ContainerStarted","Data":"c9153223047912d8f774d06aabaa0eddc39141eba2ec3a1043649258c54c5fef"} Oct 30 02:23:36.386693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:36.386509 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:36.435913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:36.435770 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" podStartSLOduration=1.435748447 podStartE2EDuration="1.435748447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:36.431566756 +0000 UTC m=+3294.934498977" watchObservedRunningTime="2026-10-30 02:23:36.435748447 +0000 UTC m=+3294.938680668" Oct 30 02:23:40.806867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:40.806819 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt"] Oct 30 02:23:40.813747 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:40.813718 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:40.831843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:40.831811 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt"] Oct 30 02:23:40.866250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:40.852944 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3d5da6-dockercfg-hd47b\"" Oct 30 02:23:40.866250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:40.856850 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3d5da6\"" Oct 30 02:23:40.978444 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:40.978392 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt\" (UID: \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:40.978880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:40.978854 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqdq\" (UniqueName: \"kubernetes.io/projected/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-kube-api-access-tcqdq\") pod \"payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt\" (UID: \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:41.081241 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:41.081147 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqdq\" (UniqueName: \"kubernetes.io/projected/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-kube-api-access-tcqdq\") pod \"payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt\" (UID: \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:41.081476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:41.081457 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt\" (UID: \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:41.082258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:41.082239 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt\" (UID: \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:41.110384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:41.110340 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqdq\" (UniqueName: \"kubernetes.io/projected/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-kube-api-access-tcqdq\") pod \"payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt\" (UID: \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:41.134341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:41.134307 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:41.454616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:41.454587 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt"] Oct 30 02:23:41.455419 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:23:41.455388 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceeb8f86_3769_42a2_8bcb_5a121ff0ec06.slice/crio-e808e3e761d9f516878c6ba1f5d2b0c921b3059518c74a437f6b4e69b8695811 WatchSource:0}: Error finding container e808e3e761d9f516878c6ba1f5d2b0c921b3059518c74a437f6b4e69b8695811: Status 404 returned error can't find the container with id e808e3e761d9f516878c6ba1f5d2b0c921b3059518c74a437f6b4e69b8695811 Oct 30 02:23:42.449256 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:42.449208 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" event={"ID":"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06","Type":"ContainerStarted","Data":"150ad4d584d538fa4d8c786c8aa8bad6793a101c579c4ed9a81dd398d5b7cf02"} Oct 30 02:23:42.449838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:42.449333 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" event={"ID":"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06","Type":"ContainerStarted","Data":"e808e3e761d9f516878c6ba1f5d2b0c921b3059518c74a437f6b4e69b8695811"} Oct 30 02:23:42.449838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:42.449488 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:42.496364 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:42.496294 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" podStartSLOduration=2.49627643 podStartE2EDuration="2.49627643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:42.493701054 +0000 UTC m=+3300.996633276" watchObservedRunningTime="2026-10-30 02:23:42.49627643 +0000 UTC m=+3300.999208705" Oct 30 02:23:43.889116 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:43.889087 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:23:43.903947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:43.903911 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:23:43.948888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:43.948859 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:23:43.965692 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:43.965655 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:23:47.399039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:47.399005 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:23:47.631129 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:23:47.631092 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:23:53.462788 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:53.462746 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:23:56.395690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:56.395637 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:23:56.396499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:23:56.396458 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:24:17.630739 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:24:17.630697 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:24:21.040194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.040157 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7"] Oct 30 02:24:21.045711 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.045683 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.072031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.071989 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7"] Oct 30 02:24:21.078929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.078897 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3d5da6-serving-cert\"" Oct 30 02:24:21.156317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.156271 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6wf\" (UniqueName: \"kubernetes.io/projected/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-kube-api-access-wr6wf\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.156540 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.156435 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.156602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.156560 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.156652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.156607 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-metrics-certs\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.267638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.267578 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.268513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.268482 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-metrics-certs\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.271873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.268973 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6wf\" (UniqueName: \"kubernetes.io/projected/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-kube-api-access-wr6wf\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.272256 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.272232 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.273256 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.273230 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.285137 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.285096 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.286890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.286853 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-metrics-certs\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.299961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.299870 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6wf\" (UniqueName: \"kubernetes.io/projected/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-kube-api-access-wr6wf\") pod \"maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.367160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.367120 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:21.685629 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:24:21.685590 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2365bd2_1955_46d8_ae7d_4cf401fe8a9a.slice/crio-dd3c13db1dddc2a0e9abad0d7d88c29840e008278d930c228e9dfd61def6da1f WatchSource:0}: Error finding container dd3c13db1dddc2a0e9abad0d7d88c29840e008278d930c228e9dfd61def6da1f: Status 404 returned error can't find the container with id dd3c13db1dddc2a0e9abad0d7d88c29840e008278d930c228e9dfd61def6da1f Oct 30 02:24:21.687012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.686981 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7"] Oct 30 02:24:21.915193 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:21.915158 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" event={"ID":"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a","Type":"ContainerStarted","Data":"dd3c13db1dddc2a0e9abad0d7d88c29840e008278d930c228e9dfd61def6da1f"} Oct 30 02:24:22.925388 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:22.925334 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" event={"ID":"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a","Type":"ContainerStarted","Data":"b793d62276258bba3a73b0f5587369e5cc2caaddab1cb0c3814fb3415ac16e86"} Oct 30 02:24:22.925913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:22.925415 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:22.973510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:22.973452 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" podStartSLOduration=1.595462081 podStartE2EDuration="1.97343648s" totalImagesPullingTime="377.974399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:24:22.971038335 +0000 UTC m=+3341.473970554" watchObservedRunningTime="2026-10-30 02:24:22.97343648 +0000 UTC m=+3341.476368700" Oct 30 02:24:28.939476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:24:28.939441 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:24:47.631967 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:24:47.631915 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:25:11.393956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:11.393719 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2wbd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:25:11.393956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:11.393816 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:25:17.631813 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:17.631767 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:25:20.094693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.094646 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2"] Oct 30 02:25:20.100087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.100055 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.120991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.120949 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2"] Oct 30 02:25:20.130361 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.130159 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"kube-root-ca.crt\"" Oct 30 02:25:20.130361 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.130192 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"openshift-service-ca.crt\"" Oct 30 02:25:20.130361 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.130277 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"routing-overlay\"" Oct 30 02:25:20.130361 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.130321 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"payload-processing-external-model-plugins-e2e-extmod-b-000edd\"" Oct 30 02:25:20.132257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.132232 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"agc-cred-a71d0b7b9515\"" Oct 30 02:25:20.132257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.132229 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"agc-cred-75862bf33366\"" Oct 30 02:25:20.132474 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.132238 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"payload-processing-external-model-e2e--74e737ad-dockercfg-mbn24\"" Oct 30 02:25:20.167273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.167222 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.167504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.167436 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.167625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.167587 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr9tp\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-kube-api-access-pr9tp\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.167765 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.167739 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.269214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.269174 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.269418 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.269384 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.269512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.269495 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr9tp\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-kube-api-access-pr9tp\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.269642 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.269623 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.270153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.270115 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.270288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.270236 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.272332 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.272297 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.297309 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.297269 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr9tp\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-kube-api-access-pr9tp\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.416942 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.416830 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:20.738008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.737976 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2"] Oct 30 02:25:20.738328 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:25:20.738306 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dd61e3_e9b2_41ed_b448_d7a97af6f1f2.slice/crio-3b4a01b78f1b1d5a3dc84f49ff3321e88bf50dee8fc280e31f73c4bbac5b4eb2 WatchSource:0}: Error finding container 3b4a01b78f1b1d5a3dc84f49ff3321e88bf50dee8fc280e31f73c4bbac5b4eb2: Status 404 returned error can't find the container with id 3b4a01b78f1b1d5a3dc84f49ff3321e88bf50dee8fc280e31f73c4bbac5b4eb2 Oct 30 02:25:20.841023 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.840982 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52"] Oct 30 02:25:20.846316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.846288 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:20.864338 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.864268 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"openshift-service-ca.crt\"" Oct 30 02:25:20.864338 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.864267 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"agc-cred-6abdfa144479\"" Oct 30 02:25:20.874208 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.874162 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"agc-cred-3c46743a09ae\"" Oct 30 02:25:20.877660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.877624 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"routing-overlay\"" Oct 30 02:25:20.877841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.877771 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"kube-root-ca.crt\"" Oct 30 02:25:20.877905 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.877850 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0ce950\"" Oct 30 02:25:20.877905 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.877863 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"payload-processing-external-model-e2e--c1fccfb3-dockercfg-fk79b\"" Oct 30 02:25:20.881085 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.881049 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52"] Oct 30 02:25:20.983058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.983015 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmlt\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-kube-api-access-2jmlt\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:20.983265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.983127 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:20.983265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.983210 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:20.983372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:20.983358 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.085113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.085010 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmlt\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-kube-api-access-2jmlt\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.085113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.085067 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.085369 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.085120 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.085369 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.085211 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.085981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.085955 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.086121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.086102 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.087757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.087741 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.110380 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.110345 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmlt\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-kube-api-access-2jmlt\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.163181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.163137 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:21.490017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.489981 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52"] Oct 30 02:25:21.490764 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:25:21.490733 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f3940f_2d6e_4527_bb54_3c48ee35c702.slice/crio-f56ab57bb935538d2fec7d2e45ec18806350545c400ef3f2362c99bfe1651dfe WatchSource:0}: Error finding container f56ab57bb935538d2fec7d2e45ec18806350545c400ef3f2362c99bfe1651dfe: Status 404 returned error can't find the container with id f56ab57bb935538d2fec7d2e45ec18806350545c400ef3f2362c99bfe1651dfe Oct 30 02:25:21.539882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.539843 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" event={"ID":"a2f3940f-2d6e-4527-bb54-3c48ee35c702","Type":"ContainerStarted","Data":"f56ab57bb935538d2fec7d2e45ec18806350545c400ef3f2362c99bfe1651dfe"} Oct 30 02:25:21.541514 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.541481 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" event={"ID":"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2","Type":"ContainerStarted","Data":"fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819"} Oct 30 02:25:21.541514 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.541543 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" event={"ID":"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2","Type":"ContainerStarted","Data":"3b4a01b78f1b1d5a3dc84f49ff3321e88bf50dee8fc280e31f73c4bbac5b4eb2"} Oct 30 02:25:21.541779 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.541573 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:21.590761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:21.590711 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" podStartSLOduration=1.590695529 podStartE2EDuration="1.590695529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:21.586380182 +0000 UTC m=+3400.089312404" watchObservedRunningTime="2026-10-30 02:25:21.590695529 +0000 UTC m=+3400.093627749" Oct 30 02:25:22.520066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.520018 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2"] Oct 30 02:25:22.526371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.526331 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:22.548926 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.548884 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2"] Oct 30 02:25:22.570042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.570000 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d97e4c\"" Oct 30 02:25:22.582488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.582432 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" event={"ID":"a2f3940f-2d6e-4527-bb54-3c48ee35c702","Type":"ContainerStarted","Data":"81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352"} Oct 30 02:25:22.582736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.582610 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:22.606485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.606431 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81adac-fa2a-4546-aed3-3682ced6fa8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2\" (UID: \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:22.606708 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.606575 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2hg\" (UniqueName: \"kubernetes.io/projected/8e81adac-fa2a-4546-aed3-3682ced6fa8f-kube-api-access-kz2hg\") pod \"payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2\" (UID: \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:22.669217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.667980 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" podStartSLOduration=2.667958743 podStartE2EDuration="2.667958743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:22.66751502 +0000 UTC m=+3401.170447241" watchObservedRunningTime="2026-10-30 02:25:22.667958743 +0000 UTC m=+3401.170890964" Oct 30 02:25:22.833772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.821333 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81adac-fa2a-4546-aed3-3682ced6fa8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2\" (UID: \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:22.833772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.822374 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2hg\" (UniqueName: \"kubernetes.io/projected/8e81adac-fa2a-4546-aed3-3682ced6fa8f-kube-api-access-kz2hg\") pod \"payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2\" (UID: \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:22.833772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.827076 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81adac-fa2a-4546-aed3-3682ced6fa8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2\" (UID: \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:22.854198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:22.854156 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2hg\" (UniqueName: \"kubernetes.io/projected/8e81adac-fa2a-4546-aed3-3682ced6fa8f-kube-api-access-kz2hg\") pod \"payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2\" (UID: \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:23.144082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:23.143974 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:23.479609 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:23.479565 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2"] Oct 30 02:25:23.485276 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:25:23.485244 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e81adac_fa2a_4546_aed3_3682ced6fa8f.slice/crio-0677f3c06138a4cb5eeb0bb513474ab151ee79a965ff7bcc49deabf33c382fd4 WatchSource:0}: Error finding container 0677f3c06138a4cb5eeb0bb513474ab151ee79a965ff7bcc49deabf33c382fd4: Status 404 returned error can't find the container with id 0677f3c06138a4cb5eeb0bb513474ab151ee79a965ff7bcc49deabf33c382fd4 Oct 30 02:25:23.708317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:23.707738 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" event={"ID":"8e81adac-fa2a-4546-aed3-3682ced6fa8f","Type":"ContainerStarted","Data":"8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7"} Oct 30 02:25:23.708317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:23.707803 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" event={"ID":"8e81adac-fa2a-4546-aed3-3682ced6fa8f","Type":"ContainerStarted","Data":"0677f3c06138a4cb5eeb0bb513474ab151ee79a965ff7bcc49deabf33c382fd4"} Oct 30 02:25:23.708317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:23.707854 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:23.755286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:23.755173 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" podStartSLOduration=1.755155335 podStartE2EDuration="1.755155335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:23.751156536 +0000 UTC m=+3402.254088775" watchObservedRunningTime="2026-10-30 02:25:23.755155335 +0000 UTC m=+3402.258087564" Oct 30 02:25:23.872993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:23.872896 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-96b798988-vwbd2"] Oct 30 02:25:28.874140 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:28.874096 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" podUID="dce2ded4-b790-40da-b518-cfb94db2760d" containerName="payload-processing" containerID="cri-o://2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0" gracePeriod=25 Oct 30 02:25:29.210028 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.209995 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:25:29.389987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.389944 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-provider-credentials\") pod \"dce2ded4-b790-40da-b518-cfb94db2760d\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " Oct 30 02:25:29.390180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.390155 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-plugins-config-volume\") pod \"dce2ded4-b790-40da-b518-cfb94db2760d\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " Oct 30 02:25:29.390270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.390242 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-routing-overlay-volume\") pod \"dce2ded4-b790-40da-b518-cfb94db2760d\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " Oct 30 02:25:29.390391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.390377 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-kube-api-access-v58pv\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-kube-api-access-v58pv\") pod \"dce2ded4-b790-40da-b518-cfb94db2760d\" (UID: \"dce2ded4-b790-40da-b518-cfb94db2760d\") " Oct 30 02:25:29.390607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.390575 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-routing-overlay-volume" pod "dce2ded4-b790-40da-b518-cfb94db2760d" (UID: "dce2ded4-b790-40da-b518-cfb94db2760d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:29.390691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.390613 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-plugins-config-volume" pod "dce2ded4-b790-40da-b518-cfb94db2760d" (UID: "dce2ded4-b790-40da-b518-cfb94db2760d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:29.391089 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.391068 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:25:29.391169 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.391093 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce2ded4-b790-40da-b518-cfb94db2760d-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:25:29.392366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.392344 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-provider-credentials" pod "dce2ded4-b790-40da-b518-cfb94db2760d" (UID: "dce2ded4-b790-40da-b518-cfb94db2760d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:29.392472 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.392454 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-kube-api-access-v58pv" pod "dce2ded4-b790-40da-b518-cfb94db2760d" (UID: "dce2ded4-b790-40da-b518-cfb94db2760d"). InnerVolumeSpecName "kube-api-access-v58pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:29.492947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.492897 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:25:29.492947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.492938 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v58pv\" (UniqueName: \"kubernetes.io/projected/dce2ded4-b790-40da-b518-cfb94db2760d-kube-api-access-v58pv\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:25:29.757797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.757691 5035 generic.go:356] "Generic (PLEG): container finished" podID="dce2ded4-b790-40da-b518-cfb94db2760d" containerID="2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0" exitCode=0 Oct 30 02:25:29.757797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.757742 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" Oct 30 02:25:29.757797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.757783 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" event={"ID":"dce2ded4-b790-40da-b518-cfb94db2760d","Type":"ContainerDied","Data":"2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0"} Oct 30 02:25:29.758084 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.757842 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-96b798988-vwbd2" event={"ID":"dce2ded4-b790-40da-b518-cfb94db2760d","Type":"ContainerDied","Data":"c9153223047912d8f774d06aabaa0eddc39141eba2ec3a1043649258c54c5fef"} Oct 30 02:25:29.758084 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.757870 5035 scope.go:122] "RemoveContainer" containerID="2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0" Oct 30 02:25:29.769389 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.769362 5035 scope.go:122] "RemoveContainer" containerID="2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0" Oct 30 02:25:29.769751 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:29.769721 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0\": container with ID starting with 2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0 not found: ID does not exist" containerID="2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0" Oct 30 02:25:29.769890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.769765 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0"} err="failed to get container status \"2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0\": rpc error: code = NotFound desc = could not find container \"2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0\": container with ID starting with 2dd1c1f88c9fdd64dd78907f92030a94009d8725962adcc57ca61e2ea131d0a0 not found: ID does not exist" Oct 30 02:25:29.816577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.816519 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-96b798988-vwbd2"] Oct 30 02:25:29.829882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:29.829840 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-96b798988-vwbd2"] Oct 30 02:25:31.645915 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:31.645877 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce2ded4-b790-40da-b518-cfb94db2760d" path="/var/lib/kubelet/pods/dce2ded4-b790-40da-b518-cfb94db2760d/volumes" Oct 30 02:25:32.362706 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.362663 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt"] Oct 30 02:25:32.363558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.363541 5035 cpu_manager.go:414] "containerMap: removing container" podUID="dce2ded4-b790-40da-b518-cfb94db2760d" containerName="payload-processing" Oct 30 02:25:32.363625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.363559 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce2ded4-b790-40da-b518-cfb94db2760d" containerName="payload-processing" Oct 30 02:25:32.363724 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.363714 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce2ded4-b790-40da-b518-cfb94db2760d" containerName="payload-processing" Oct 30 02:25:32.384109 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.384065 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv"] Oct 30 02:25:32.387253 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.387207 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:32.390966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.390923 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt"] Oct 30 02:25:32.391161 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.391063 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:32.401686 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.401649 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv"] Oct 30 02:25:32.405012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.404983 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1fddb7-dockercfg-mgkxv\"" Oct 30 02:25:32.414689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.414650 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1fddb7\"" Oct 30 02:25:32.443069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.443007 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmv5m\" (UniqueName: \"kubernetes.io/projected/8a3c62f3-e816-4358-b53e-6b9698c66531-kube-api-access-zmv5m\") pod \"payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv\" (UID: \"8a3c62f3-e816-4358-b53e-6b9698c66531\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:32.443069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.443063 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89a119f-d530-4b93-b381-eb23753d22ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt\" (UID: \"d89a119f-d530-4b93-b381-eb23753d22ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:32.443384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.443188 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3c62f3-e816-4358-b53e-6b9698c66531-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv\" (UID: \"8a3c62f3-e816-4358-b53e-6b9698c66531\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:32.443384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.443330 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r87gm\" (UniqueName: \"kubernetes.io/projected/d89a119f-d530-4b93-b381-eb23753d22ee-kube-api-access-r87gm\") pod \"payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt\" (UID: \"d89a119f-d530-4b93-b381-eb23753d22ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:32.583327 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.583207 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3c62f3-e816-4358-b53e-6b9698c66531-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv\" (UID: \"8a3c62f3-e816-4358-b53e-6b9698c66531\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:32.583799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.583769 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r87gm\" (UniqueName: \"kubernetes.io/projected/d89a119f-d530-4b93-b381-eb23753d22ee-kube-api-access-r87gm\") pod \"payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt\" (UID: \"d89a119f-d530-4b93-b381-eb23753d22ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:32.584261 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.584237 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmv5m\" (UniqueName: \"kubernetes.io/projected/8a3c62f3-e816-4358-b53e-6b9698c66531-kube-api-access-zmv5m\") pod \"payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv\" (UID: \"8a3c62f3-e816-4358-b53e-6b9698c66531\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:32.584415 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.584400 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89a119f-d530-4b93-b381-eb23753d22ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt\" (UID: \"d89a119f-d530-4b93-b381-eb23753d22ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:32.590749 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.589143 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3c62f3-e816-4358-b53e-6b9698c66531-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv\" (UID: \"8a3c62f3-e816-4358-b53e-6b9698c66531\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:32.598180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.596258 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89a119f-d530-4b93-b381-eb23753d22ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt\" (UID: \"d89a119f-d530-4b93-b381-eb23753d22ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:32.611329 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.611289 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:25:32.615789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.615700 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmv5m\" (UniqueName: \"kubernetes.io/projected/8a3c62f3-e816-4358-b53e-6b9698c66531-kube-api-access-zmv5m\") pod \"payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv\" (UID: \"8a3c62f3-e816-4358-b53e-6b9698c66531\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:32.631871 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.631473 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r87gm\" (UniqueName: \"kubernetes.io/projected/d89a119f-d530-4b93-b381-eb23753d22ee-kube-api-access-r87gm\") pod \"payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt\" (UID: \"d89a119f-d530-4b93-b381-eb23753d22ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:32.720362 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.720324 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:32.729092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:32.729054 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:33.059686 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:25:33.059647 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89a119f_d530_4b93_b381_eb23753d22ee.slice/crio-cd7eb76b440820a5874ef314906915e06465692fab9632e789b157654bc4a27e WatchSource:0}: Error finding container cd7eb76b440820a5874ef314906915e06465692fab9632e789b157654bc4a27e: Status 404 returned error can't find the container with id cd7eb76b440820a5874ef314906915e06465692fab9632e789b157654bc4a27e Oct 30 02:25:33.059686 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.059664 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt"] Oct 30 02:25:33.088249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.088215 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv"] Oct 30 02:25:33.089607 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:25:33.089573 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3c62f3_e816_4358_b53e_6b9698c66531.slice/crio-894a14506471acca25182954f1a092708b17f907ba3bcf0165154521b47787fb WatchSource:0}: Error finding container 894a14506471acca25182954f1a092708b17f907ba3bcf0165154521b47787fb: Status 404 returned error can't find the container with id 894a14506471acca25182954f1a092708b17f907ba3bcf0165154521b47787fb Oct 30 02:25:33.710417 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.710383 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:25:33.801081 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.801043 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" event={"ID":"d89a119f-d530-4b93-b381-eb23753d22ee","Type":"ContainerStarted","Data":"e05c913fc8df4404d4fb354be1627ce00354ff05f8379defcc0b12aab429855e"} Oct 30 02:25:33.801617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.801112 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" event={"ID":"d89a119f-d530-4b93-b381-eb23753d22ee","Type":"ContainerStarted","Data":"cd7eb76b440820a5874ef314906915e06465692fab9632e789b157654bc4a27e"} Oct 30 02:25:33.801617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.801282 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:33.804567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.804516 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" event={"ID":"8a3c62f3-e816-4358-b53e-6b9698c66531","Type":"ContainerStarted","Data":"ec93caf82af20ffa1de85259a8a1c35c50c2004fdfae3dca0aefab2b3c17f8a7"} Oct 30 02:25:33.804749 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.804592 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" event={"ID":"8a3c62f3-e816-4358-b53e-6b9698c66531","Type":"ContainerStarted","Data":"894a14506471acca25182954f1a092708b17f907ba3bcf0165154521b47787fb"} Oct 30 02:25:33.804749 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.804633 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:33.863644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.863576 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" podStartSLOduration=1.863555786 podStartE2EDuration="1.863555786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:33.861059741 +0000 UTC m=+3412.363991993" watchObservedRunningTime="2026-10-30 02:25:33.863555786 +0000 UTC m=+3412.366488008" Oct 30 02:25:33.914049 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:33.913996 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" podStartSLOduration=1.91398103 podStartE2EDuration="1.91398103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:33.913923106 +0000 UTC m=+3412.416855327" watchObservedRunningTime="2026-10-30 02:25:33.91398103 +0000 UTC m=+3412.416913279" Oct 30 02:25:35.747791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:35.747742 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:25:36.488381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.488332 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl"] Oct 30 02:25:36.494655 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.494623 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.513350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.513296 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"openshift-service-ca.crt\"" Oct 30 02:25:36.513350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.513312 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"payload-processing-external-model-plugins-e2e-extmod-b-93ec46\"" Oct 30 02:25:36.513350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.513331 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"routing-overlay\"" Oct 30 02:25:36.513350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.513309 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"payload-processing-external-model-e2e--dd78327c-dockercfg-8hk6c\"" Oct 30 02:25:36.513664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.513370 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"agc-cred-8974cf53f130\"" Oct 30 02:25:36.536576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.527180 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"kube-root-ca.crt\"" Oct 30 02:25:36.536576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.532594 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl"] Oct 30 02:25:36.562172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.562101 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.562382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.562349 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.562588 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.562515 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkkck\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-kube-api-access-lkkck\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.562702 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.562597 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.663839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.663802 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.664075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.663996 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkkck\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-kube-api-access-lkkck\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.664075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.664053 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.664200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.664137 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.664253 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:36.664230 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-93ec46/agc-cred-8974cf53f130: secret "agc-cred-8974cf53f130" not found Oct 30 02:25:36.664302 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:36.664259 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl: secret "agc-cred-8974cf53f130" not found Oct 30 02:25:36.664351 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:36.664326 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials podName:b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b nodeName:}" failed. No retries permitted until 2026-10-30 02:25:37.164302578 +0000 UTC m=+3415.667234776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" (UID: "b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b") : secret "agc-cred-8974cf53f130" not found Oct 30 02:25:36.664891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.664862 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.664891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.664886 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:36.691044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:36.691003 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkkck\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-kube-api-access-lkkck\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:37.189094 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:37.189048 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:37.189519 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:37.189242 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-93ec46/agc-cred-8974cf53f130: secret "agc-cred-8974cf53f130" not found Oct 30 02:25:37.189519 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:37.189266 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl: secret "agc-cred-8974cf53f130" not found Oct 30 02:25:37.189519 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:37.189324 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials podName:b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b nodeName:}" failed. No retries permitted until 2026-10-30 02:25:38.189310072 +0000 UTC m=+3416.692242270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" (UID: "b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b") : secret "agc-cred-8974cf53f130" not found Oct 30 02:25:37.909594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:37.909544 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl"] Oct 30 02:25:37.910024 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:37.909993 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" podUID="b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b" Oct 30 02:25:37.967436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:37.967400 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:37.978465 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:37.978435 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:38.112814 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.112774 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-plugins-config-volume\") pod \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " Oct 30 02:25:38.113069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.113055 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-kube-api-access-lkkck\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-kube-api-access-lkkck\") pod \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " Oct 30 02:25:38.113153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.113108 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-routing-overlay-volume\") pod \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " Oct 30 02:25:38.113194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.113173 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-plugins-config-volume" pod "b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b" (UID: "b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:38.113522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.113500 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-routing-overlay-volume" pod "b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b" (UID: "b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:38.113789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.113773 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:25:38.113834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.113792 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:25:38.115406 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.115386 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-kube-api-access-lkkck" pod "b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b" (UID: "b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b"). InnerVolumeSpecName "kube-api-access-lkkck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:38.216243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.216177 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl\" (UID: \"b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:38.216979 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:38.216450 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-93ec46/agc-cred-8974cf53f130: secret "agc-cred-8974cf53f130" not found Oct 30 02:25:38.217075 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:38.216990 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl: secret "agc-cred-8974cf53f130" not found Oct 30 02:25:38.217152 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:38.217091 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials podName:b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b nodeName:}" failed. No retries permitted until 2026-10-30 02:25:40.217064428 +0000 UTC m=+3418.719996650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" (UID: "b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b") : secret "agc-cred-8974cf53f130" not found Oct 30 02:25:38.217438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.217400 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkkck\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-kube-api-access-lkkck\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:25:38.976139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:38.976104 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl" Oct 30 02:25:39.071753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:39.071704 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl"] Oct 30 02:25:39.077497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:39.077461 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66cxdbl"] Oct 30 02:25:39.241015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:39.240909 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:25:39.646090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:39.646005 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b" path="/var/lib/kubelet/pods/b6bd9a6d-0ef1-4f4a-9de1-57dd362cca5b/volumes" Oct 30 02:25:44.816065 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:44.816031 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:25:44.816590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:25:44.816117 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:25:47.632521 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:25:47.632481 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:26:07.460572 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.460520 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p"] Oct 30 02:26:07.466930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.466900 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.492010 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.491973 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npks6\" (UniqueName: \"kubernetes.io/projected/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-kube-api-access-npks6\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.492483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.492458 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.495082 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.495054 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-maas-api-tls\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.496676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.496647 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-metrics-certs\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.503900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.503862 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d97e4c-serving-cert\"" Oct 30 02:26:07.512367 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.512331 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p"] Oct 30 02:26:07.598362 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.598308 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npks6\" (UniqueName: \"kubernetes.io/projected/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-kube-api-access-npks6\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.598613 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.598438 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.598613 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.598488 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-maas-api-tls\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.598700 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.598661 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-metrics-certs\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.599247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.599210 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.601298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.601271 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-maas-api-tls\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.601412 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.601321 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-metrics-certs\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.623640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.623595 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npks6\" (UniqueName: \"kubernetes.io/projected/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-kube-api-access-npks6\") pod \"maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:07.785778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:07.785683 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:08.104737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:08.104706 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p"] Oct 30 02:26:08.105245 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:26:08.105216 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757b9410_ecd5_4cb2_88c3_5aec3f3eff8c.slice/crio-ed9108bb6f483acfc64f0f54685a11b76315161e604a8d1910f41e2a18c91f28 WatchSource:0}: Error finding container ed9108bb6f483acfc64f0f54685a11b76315161e604a8d1910f41e2a18c91f28: Status 404 returned error can't find the container with id ed9108bb6f483acfc64f0f54685a11b76315161e604a8d1910f41e2a18c91f28 Oct 30 02:26:08.106904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:08.106880 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:26:08.287237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:08.287191 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" event={"ID":"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c","Type":"ContainerStarted","Data":"ed9108bb6f483acfc64f0f54685a11b76315161e604a8d1910f41e2a18c91f28"} Oct 30 02:26:09.299351 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:09.299299 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" event={"ID":"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c","Type":"ContainerStarted","Data":"6be532caf078b7039d3f5ee9adf83ffe291b380569ace64a86af03fdbd0a168c"} Oct 30 02:26:09.300432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:09.299414 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:09.349513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:09.349449 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" podStartSLOduration=1.979151951 podStartE2EDuration="2.349431419s" totalImagesPullingTime="370.279468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:26:09.347692344 +0000 UTC m=+3447.850624566" watchObservedRunningTime="2026-10-30 02:26:09.349431419 +0000 UTC m=+3447.852363683" Oct 30 02:26:15.313782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:15.313743 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:26:17.632639 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:26:17.632596 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:26:20.877056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:20.877018 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj"] Oct 30 02:26:20.882437 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:20.882404 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:20.899236 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:20.899189 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1fddb7-serving-cert\"" Oct 30 02:26:20.910438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:20.910398 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj"] Oct 30 02:26:21.032315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.032273 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4lk6\" (UniqueName: \"kubernetes.io/projected/12564f35-a37b-4dd7-befd-91ee03e6b39d-kube-api-access-m4lk6\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.032315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.032317 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.032666 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.032437 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-metrics-certs\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.032782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.032759 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12564f35-a37b-4dd7-befd-91ee03e6b39d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.141043 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.140946 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4lk6\" (UniqueName: \"kubernetes.io/projected/12564f35-a37b-4dd7-befd-91ee03e6b39d-kube-api-access-m4lk6\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.141328 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.141299 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.141574 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.141548 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-metrics-certs\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.143318 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.142023 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12564f35-a37b-4dd7-befd-91ee03e6b39d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.144484 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.144454 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12564f35-a37b-4dd7-befd-91ee03e6b39d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.148953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.148924 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.149430 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.149386 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-metrics-certs\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.177979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.173668 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4lk6\" (UniqueName: \"kubernetes.io/projected/12564f35-a37b-4dd7-befd-91ee03e6b39d-kube-api-access-m4lk6\") pod \"maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.214927 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.214885 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:21.531102 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:21.531065 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj"] Oct 30 02:26:21.531762 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:26:21.531734 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12564f35_a37b_4dd7_befd_91ee03e6b39d.slice/crio-3713843187f0b69e5f9a24e7fc43647bb513349c45ce955afba35e0d64a1a5ae WatchSource:0}: Error finding container 3713843187f0b69e5f9a24e7fc43647bb513349c45ce955afba35e0d64a1a5ae: Status 404 returned error can't find the container with id 3713843187f0b69e5f9a24e7fc43647bb513349c45ce955afba35e0d64a1a5ae Oct 30 02:26:22.411617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:22.411556 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" event={"ID":"12564f35-a37b-4dd7-befd-91ee03e6b39d","Type":"ContainerStarted","Data":"1e82963a423512d8876a86feb58a1336a57658ba0efbf098cf49e398a3406659"} Oct 30 02:26:22.411617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:22.411625 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" event={"ID":"12564f35-a37b-4dd7-befd-91ee03e6b39d","Type":"ContainerStarted","Data":"3713843187f0b69e5f9a24e7fc43647bb513349c45ce955afba35e0d64a1a5ae"} Oct 30 02:26:22.412126 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:22.411650 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:22.462286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:22.462224 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" podStartSLOduration=2.092144814 podStartE2EDuration="2.462206465s" totalImagesPullingTime="370.061651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:26:22.458744502 +0000 UTC m=+3460.961676734" watchObservedRunningTime="2026-10-30 02:26:22.462206465 +0000 UTC m=+3460.965138722" Oct 30 02:26:28.428755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:28.428668 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:26:43.955188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:26:43.955149 5035 scope.go:122] "RemoveContainer" containerID="29dc9bbac2d1d100ebe1d2994c1ab644f691ec83fba9471c9acd678426773bef" Oct 30 02:26:47.633962 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:26:47.633922 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:27:08.329973 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.329930 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j"] Oct 30 02:27:08.335477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.335441 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:08.356841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.356795 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j"] Oct 30 02:27:08.364217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.364179 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4593af\"" Oct 30 02:27:08.364217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.364209 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4593af-dockercfg-x6dzw\"" Oct 30 02:27:08.444587 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.444542 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4qw\" (UniqueName: \"kubernetes.io/projected/4d8b92e3-f14b-4575-bdde-de31e8696afa-kube-api-access-2g4qw\") pod \"payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j\" (UID: \"4d8b92e3-f14b-4575-bdde-de31e8696afa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:08.445060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.445029 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8b92e3-f14b-4575-bdde-de31e8696afa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j\" (UID: \"4d8b92e3-f14b-4575-bdde-de31e8696afa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:08.557618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.555155 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4qw\" (UniqueName: \"kubernetes.io/projected/4d8b92e3-f14b-4575-bdde-de31e8696afa-kube-api-access-2g4qw\") pod \"payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j\" (UID: \"4d8b92e3-f14b-4575-bdde-de31e8696afa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:08.557618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.555417 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8b92e3-f14b-4575-bdde-de31e8696afa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j\" (UID: \"4d8b92e3-f14b-4575-bdde-de31e8696afa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:08.557618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.556504 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8b92e3-f14b-4575-bdde-de31e8696afa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j\" (UID: \"4d8b92e3-f14b-4575-bdde-de31e8696afa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:08.604141 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.600925 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4qw\" (UniqueName: \"kubernetes.io/projected/4d8b92e3-f14b-4575-bdde-de31e8696afa-kube-api-access-2g4qw\") pod \"payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j\" (UID: \"4d8b92e3-f14b-4575-bdde-de31e8696afa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:08.654062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.654026 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:08.713996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.713957 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst"] Oct 30 02:27:08.714213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.714190 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" containerID="cri-o://67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f" gracePeriod=30 Oct 30 02:27:08.981383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:08.981338 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j"] Oct 30 02:27:08.981933 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:27:08.981906 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8b92e3_f14b_4575_bdde_de31e8696afa.slice/crio-076d3fd0a7bef001cd378d9ca88851611a012a81b1305120e684f335adf61790 WatchSource:0}: Error finding container 076d3fd0a7bef001cd378d9ca88851611a012a81b1305120e684f335adf61790: Status 404 returned error can't find the container with id 076d3fd0a7bef001cd378d9ca88851611a012a81b1305120e684f335adf61790 Oct 30 02:27:09.995626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:09.995580 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" event={"ID":"4d8b92e3-f14b-4575-bdde-de31e8696afa","Type":"ContainerStarted","Data":"ed8edf3fc2438a744c33cf195862863b55ebf4183034868274a4e4515da81b49"} Oct 30 02:27:09.996073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:09.995656 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" event={"ID":"4d8b92e3-f14b-4575-bdde-de31e8696afa","Type":"ContainerStarted","Data":"076d3fd0a7bef001cd378d9ca88851611a012a81b1305120e684f335adf61790"} Oct 30 02:27:09.996073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:09.995692 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:10.046267 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:10.046210 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" podStartSLOduration=2.046194589 podStartE2EDuration="2.046194589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:10.042993735 +0000 UTC m=+3508.545925957" watchObservedRunningTime="2026-10-30 02:27:10.046194589 +0000 UTC m=+3508.549126808" Oct 30 02:27:10.403342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:10.403253 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:27:10.403342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:10.403324 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:27:14.027862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.027817 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 02:27:14.032469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.032435 5035 generic.go:356] "Generic (PLEG): container finished" podID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerID="67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f" exitCode=0 Oct 30 02:27:14.032616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.032480 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" Oct 30 02:27:14.032616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.032512 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" event={"ID":"71474faf-7414-4bbb-8d8d-592e30db5efc","Type":"ContainerDied","Data":"67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f"} Oct 30 02:27:14.032616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.032589 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst" event={"ID":"71474faf-7414-4bbb-8d8d-592e30db5efc","Type":"ContainerDied","Data":"de07b3622ba41ce23cf5aabc9a8d985682848cef667e4bc980c3cd7229a20b6c"} Oct 30 02:27:14.032742 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.032625 5035 scope.go:122] "RemoveContainer" containerID="67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f" Oct 30 02:27:14.045018 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.044995 5035 scope.go:122] "RemoveContainer" containerID="67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f" Oct 30 02:27:14.045388 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:14.045365 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f\": container with ID starting with 67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f not found: ID does not exist" containerID="67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f" Oct 30 02:27:14.045470 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.045403 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f"} err="failed to get container status \"67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f\": rpc error: code = NotFound desc = could not find container \"67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f\": container with ID starting with 67d5a10be507592303b9153cbf28fcc7c5a67b923491556435d070e7b064bc6f not found: ID does not exist" Oct 30 02:27:14.179304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179260 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-credential-socket\") pod \"71474faf-7414-4bbb-8d8d-592e30db5efc\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " Oct 30 02:27:14.179508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179327 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-socket\") pod \"71474faf-7414-4bbb-8d8d-592e30db5efc\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " Oct 30 02:27:14.179508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179403 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-envoy\") pod \"71474faf-7414-4bbb-8d8d-592e30db5efc\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " Oct 30 02:27:14.179508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179454 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-data\") pod \"71474faf-7414-4bbb-8d8d-592e30db5efc\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " Oct 30 02:27:14.179723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179512 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-podinfo\") pod \"71474faf-7414-4bbb-8d8d-592e30db5efc\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " Oct 30 02:27:14.179723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179605 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-certs\") pod \"71474faf-7414-4bbb-8d8d-592e30db5efc\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " Oct 30 02:27:14.179723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179673 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-credential-socket" pod "71474faf-7414-4bbb-8d8d-592e30db5efc" (UID: "71474faf-7414-4bbb-8d8d-592e30db5efc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:14.179723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179687 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-socket" pod "71474faf-7414-4bbb-8d8d-592e30db5efc" (UID: "71474faf-7414-4bbb-8d8d-592e30db5efc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:14.179952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179795 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-data" pod "71474faf-7414-4bbb-8d8d-592e30db5efc" (UID: "71474faf-7414-4bbb-8d8d-592e30db5efc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:14.179952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179863 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71474faf-7414-4bbb-8d8d-592e30db5efc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71474faf-7414-4bbb-8d8d-592e30db5efc-istiod-ca-cert\") pod \"71474faf-7414-4bbb-8d8d-592e30db5efc\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " Oct 30 02:27:14.179952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179901 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-certs" pod "71474faf-7414-4bbb-8d8d-592e30db5efc" (UID: "71474faf-7414-4bbb-8d8d-592e30db5efc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:14.179952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179928 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-token\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-token\") pod \"71474faf-7414-4bbb-8d8d-592e30db5efc\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " Oct 30 02:27:14.180183 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.179982 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-kube-api-access-qfnzr\") pod \"71474faf-7414-4bbb-8d8d-592e30db5efc\" (UID: \"71474faf-7414-4bbb-8d8d-592e30db5efc\") " Oct 30 02:27:14.180250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.180205 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71474faf-7414-4bbb-8d8d-592e30db5efc-istiod-ca-cert" pod "71474faf-7414-4bbb-8d8d-592e30db5efc" (UID: "71474faf-7414-4bbb-8d8d-592e30db5efc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:14.180898 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.180870 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:14.181044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.180900 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:14.181044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.180917 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71474faf-7414-4bbb-8d8d-592e30db5efc-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:14.181044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.180930 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:14.181044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.180944 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:14.182014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.181983 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-podinfo" pod "71474faf-7414-4bbb-8d8d-592e30db5efc" (UID: "71474faf-7414-4bbb-8d8d-592e30db5efc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:27:14.182135 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.181992 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-envoy" pod "71474faf-7414-4bbb-8d8d-592e30db5efc" (UID: "71474faf-7414-4bbb-8d8d-592e30db5efc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:14.182202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.182160 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-token" pod "71474faf-7414-4bbb-8d8d-592e30db5efc" (UID: "71474faf-7414-4bbb-8d8d-592e30db5efc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:14.182306 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.182289 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-kube-api-access-qfnzr" pod "71474faf-7414-4bbb-8d8d-592e30db5efc" (UID: "71474faf-7414-4bbb-8d8d-592e30db5efc"). InnerVolumeSpecName "kube-api-access-qfnzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:14.282570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.282458 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:14.282570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.282500 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/71474faf-7414-4bbb-8d8d-592e30db5efc-kube-api-access-qfnzr\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:14.282570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.282510 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:14.282570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.282521 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71474faf-7414-4bbb-8d8d-592e30db5efc-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:14.390865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.390821 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst"] Oct 30 02:27:14.398739 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:14.398697 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-4nhst"] Oct 30 02:27:15.647191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:15.647141 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" path="/var/lib/kubelet/pods/71474faf-7414-4bbb-8d8d-592e30db5efc/volumes" Oct 30 02:27:17.634690 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:17.634648 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:27:18.956636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:18.956542 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2"] Oct 30 02:27:18.959387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:18.959352 5035 cpu_manager.go:414] "containerMap: removing container" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" Oct 30 02:27:18.959387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:18.959386 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" Oct 30 02:27:18.959701 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:18.959690 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="71474faf-7414-4bbb-8d8d-592e30db5efc" containerName="istio-proxy" Oct 30 02:27:18.966718 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:18.966688 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:18.995676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:18.995636 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2"] Oct 30 02:27:19.011792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.007836 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c75203-openshift-default-dockercfg-vptq4\"" Oct 30 02:27:19.074406 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.074362 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-envoy\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.074643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.074462 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvx5n\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-kube-api-access-xvx5n\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.074643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.074506 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-socket\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.074787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.074655 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-credential-socket\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.074787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.074682 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-token\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.074787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.074753 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-certs\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.074966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.074947 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27674de-a2e2-4e4f-82a2-00e44a555270-istiod-ca-cert\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.075090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.075075 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-podinfo\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.075171 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.075110 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-data\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.177459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.177406 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27674de-a2e2-4e4f-82a2-00e44a555270-istiod-ca-cert\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.177702 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.177560 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-podinfo\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.177702 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.177591 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-data\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.177702 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.177660 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-envoy\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.177702 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.177687 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx5n\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-kube-api-access-xvx5n\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.177941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.177716 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-socket\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.178200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.178176 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-data\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.179000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.178973 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27674de-a2e2-4e4f-82a2-00e44a555270-istiod-ca-cert\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.179596 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.179566 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-socket\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.179855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.179835 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-credential-socket\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.179943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.179915 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-token\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.180000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.179981 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-certs\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.180306 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.180274 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-credential-socket\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.180402 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.180316 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-certs\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.182056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.182026 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-envoy\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.186302 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.186271 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-podinfo\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.209034 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.208950 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvx5n\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-kube-api-access-xvx5n\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.213948 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.213912 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-token\") pod \"e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.326066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.326023 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:19.701519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.701482 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2"] Oct 30 02:27:19.701519 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:27:19.701511 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27674de_a2e2_4e4f_82a2_00e44a555270.slice/crio-0fe61e5677ff98c1ab2018db1f5025e068a6b2844671ffce6b5fd61f7413dd1a WatchSource:0}: Error finding container 0fe61e5677ff98c1ab2018db1f5025e068a6b2844671ffce6b5fd61f7413dd1a: Status 404 returned error can't find the container with id 0fe61e5677ff98c1ab2018db1f5025e068a6b2844671ffce6b5fd61f7413dd1a Oct 30 02:27:19.704994 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.704958 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:27:19.705096 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.705032 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:27:19.705096 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:19.705078 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:27:20.088511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:20.088461 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" event={"ID":"d27674de-a2e2-4e4f-82a2-00e44a555270","Type":"ContainerStarted","Data":"e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3"} Oct 30 02:27:20.089048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:20.088584 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" event={"ID":"d27674de-a2e2-4e4f-82a2-00e44a555270","Type":"ContainerStarted","Data":"0fe61e5677ff98c1ab2018db1f5025e068a6b2844671ffce6b5fd61f7413dd1a"} Oct 30 02:27:20.181755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:20.181682 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" podStartSLOduration=2.181661509 podStartE2EDuration="2.181661509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:20.163141372 +0000 UTC m=+3518.666073593" watchObservedRunningTime="2026-10-30 02:27:20.181661509 +0000 UTC m=+3518.684593730" Oct 30 02:27:20.327065 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:20.327017 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:20.335487 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:20.335455 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:21.010446 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:21.010405 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:27:21.100337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:21.097167 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:21.100337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:21.098520 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:27:24.622488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.622449 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf"] Oct 30 02:27:24.628273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.628241 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:24.644919 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.644862 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c75203\"" Oct 30 02:27:24.645169 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.645139 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c75203-dockercfg-wbj6l\"" Oct 30 02:27:24.659557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.656150 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf"] Oct 30 02:27:24.688422 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.688371 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxwr5\" (UniqueName: \"kubernetes.io/projected/3bf1dda1-3b78-4da2-906b-8882ffee8025-kube-api-access-pxwr5\") pod \"payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf\" (UID: \"3bf1dda1-3b78-4da2-906b-8882ffee8025\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:24.688903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.688874 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf1dda1-3b78-4da2-906b-8882ffee8025-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf\" (UID: \"3bf1dda1-3b78-4da2-906b-8882ffee8025\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:24.854551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.854491 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwr5\" (UniqueName: \"kubernetes.io/projected/3bf1dda1-3b78-4da2-906b-8882ffee8025-kube-api-access-pxwr5\") pod \"payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf\" (UID: \"3bf1dda1-3b78-4da2-906b-8882ffee8025\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:24.874607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.873339 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf1dda1-3b78-4da2-906b-8882ffee8025-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf\" (UID: \"3bf1dda1-3b78-4da2-906b-8882ffee8025\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:24.874607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.874570 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf1dda1-3b78-4da2-906b-8882ffee8025-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf\" (UID: \"3bf1dda1-3b78-4da2-906b-8882ffee8025\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:24.904548 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.903958 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwr5\" (UniqueName: \"kubernetes.io/projected/3bf1dda1-3b78-4da2-906b-8882ffee8025-kube-api-access-pxwr5\") pod \"payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf\" (UID: \"3bf1dda1-3b78-4da2-906b-8882ffee8025\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:24.948090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:24.948029 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:25.270967 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:25.270932 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf"] Oct 30 02:27:25.271729 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:27:25.271702 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf1dda1_3b78_4da2_906b_8882ffee8025.slice/crio-babb2cde694de5be9cc17a640702b6b37679149f4bd834abf7f2ee1e132a8e83 WatchSource:0}: Error finding container babb2cde694de5be9cc17a640702b6b37679149f4bd834abf7f2ee1e132a8e83: Status 404 returned error can't find the container with id babb2cde694de5be9cc17a640702b6b37679149f4bd834abf7f2ee1e132a8e83 Oct 30 02:27:26.143313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:26.143274 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" event={"ID":"3bf1dda1-3b78-4da2-906b-8882ffee8025","Type":"ContainerStarted","Data":"7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37"} Oct 30 02:27:26.143843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:26.143340 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" event={"ID":"3bf1dda1-3b78-4da2-906b-8882ffee8025","Type":"ContainerStarted","Data":"babb2cde694de5be9cc17a640702b6b37679149f4bd834abf7f2ee1e132a8e83"} Oct 30 02:27:26.143843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:26.143367 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:26.207737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:26.207673 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" podStartSLOduration=2.207656572 podStartE2EDuration="2.207656572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:26.205933168 +0000 UTC m=+3524.708865389" watchObservedRunningTime="2026-10-30 02:27:26.207656572 +0000 UTC m=+3524.710588792" Oct 30 02:27:37.154811 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:37.154773 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:27:37.876605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:37.865887 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt"] Oct 30 02:27:37.901514 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:37.898415 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:37.936777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:37.931119 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:37.936777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:37.931409 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqlg\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-kube-api-access-9kqlg\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:37.936777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:37.931474 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:37.936777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:37.931512 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:37.939550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:37.937811 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt"] Oct 30 02:27:38.041495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.041014 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqlg\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-kube-api-access-9kqlg\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:38.041751 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.041678 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:38.041820 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.041771 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:38.041956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.041930 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:38.043354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.043324 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:38.043563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.043419 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:38.050176 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.050145 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:38.071938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.071824 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqlg\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-kube-api-access-9kqlg\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:38.285394 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.280875 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:38.663486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.663450 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt"] Oct 30 02:27:38.667292 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:27:38.667250 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a53aa1_fbf4_4e68_9605_7ba41c073f0a.slice/crio-77298d8c3d51432a0cf13e006f735a0b694c5aaf0ba4ab1b9c662a45651e5847 WatchSource:0}: Error finding container 77298d8c3d51432a0cf13e006f735a0b694c5aaf0ba4ab1b9c662a45651e5847: Status 404 returned error can't find the container with id 77298d8c3d51432a0cf13e006f735a0b694c5aaf0ba4ab1b9c662a45651e5847 Oct 30 02:27:38.976900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.976864 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v"] Oct 30 02:27:38.982789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:38.982758 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.007716 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.007682 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v"] Oct 30 02:27:39.113077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.113036 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.113077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.113083 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.113311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.113219 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.113357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.113333 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th9b6\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-kube-api-access-th9b6\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.214956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.214917 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th9b6\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-kube-api-access-th9b6\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.215175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.215126 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.215175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.215157 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.215357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.215331 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.215881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.215860 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.216029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.216008 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.217929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.217903 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.240157 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.240118 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th9b6\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-kube-api-access-th9b6\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.293007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.292963 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" event={"ID":"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a","Type":"ContainerStarted","Data":"b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0"} Oct 30 02:27:39.293512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.293032 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:39.293512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.293049 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" event={"ID":"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a","Type":"ContainerStarted","Data":"77298d8c3d51432a0cf13e006f735a0b694c5aaf0ba4ab1b9c662a45651e5847"} Oct 30 02:27:39.314071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.314028 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:39.341478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.341409 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" podStartSLOduration=2.341386361 podStartE2EDuration="2.341386361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:39.33787581 +0000 UTC m=+3537.840808031" watchObservedRunningTime="2026-10-30 02:27:39.341386361 +0000 UTC m=+3537.844318584" Oct 30 02:27:39.632386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:39.632347 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v"] Oct 30 02:27:39.633513 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:27:39.633483 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f65130_417f_46b7_97ef_a68f9daf137a.slice/crio-0d8d99f3b2dbe66f5c3c1f359041e4109334e77750e3e7082350f98f1665ee30 WatchSource:0}: Error finding container 0d8d99f3b2dbe66f5c3c1f359041e4109334e77750e3e7082350f98f1665ee30: Status 404 returned error can't find the container with id 0d8d99f3b2dbe66f5c3c1f359041e4109334e77750e3e7082350f98f1665ee30 Oct 30 02:27:40.305266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:40.305221 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" event={"ID":"26f65130-417f-46b7-97ef-a68f9daf137a","Type":"ContainerStarted","Data":"d71b9462e8bb4b99b6da72f6b827e4fd34da21daaa44e135116df63174c66423"} Oct 30 02:27:40.305752 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:40.305323 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" event={"ID":"26f65130-417f-46b7-97ef-a68f9daf137a","Type":"ContainerStarted","Data":"0d8d99f3b2dbe66f5c3c1f359041e4109334e77750e3e7082350f98f1665ee30"} Oct 30 02:27:40.305752 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:40.305556 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:40.351489 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:40.351426 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" podStartSLOduration=2.35140969 podStartE2EDuration="2.35140969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:40.350182695 +0000 UTC m=+3538.853114917" watchObservedRunningTime="2026-10-30 02:27:40.35140969 +0000 UTC m=+3538.854341909" Oct 30 02:27:44.727661 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:44.727618 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-000edd/agc-cred-75862bf33366: secret "agc-cred-75862bf33366" not found Oct 30 02:27:44.727661 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:44.727664 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2: secret "agc-cred-75862bf33366" not found Oct 30 02:27:44.728292 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:44.727732 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials podName:50dd61e3-e9b2-41ed-b448-d7a97af6f1f2 nodeName:}" failed. No retries permitted until 2026-10-30 02:27:45.227710959 +0000 UTC m=+3543.730643174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" (UID: "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2") : secret "agc-cred-75862bf33366" not found Oct 30 02:27:45.010469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.010374 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs"] Oct 30 02:27:45.015704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.015678 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:45.036753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.036718 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d207e8\"" Oct 30 02:27:45.041769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.041732 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs"] Oct 30 02:27:45.133452 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.133407 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fd086c-87ea-4a37-8ab3-d909e528c75a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs\" (UID: \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:45.133717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.133686 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl66f\" (UniqueName: \"kubernetes.io/projected/f1fd086c-87ea-4a37-8ab3-d909e528c75a-kube-api-access-kl66f\") pod \"payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs\" (UID: \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:45.241921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.241854 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fd086c-87ea-4a37-8ab3-d909e528c75a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs\" (UID: \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:45.242444 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.242417 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl66f\" (UniqueName: \"kubernetes.io/projected/f1fd086c-87ea-4a37-8ab3-d909e528c75a-kube-api-access-kl66f\") pod \"payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs\" (UID: \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:45.243964 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.243936 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fd086c-87ea-4a37-8ab3-d909e528c75a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs\" (UID: \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:45.244219 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:45.244200 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-000edd/agc-cred-75862bf33366: secret "agc-cred-75862bf33366" not found Oct 30 02:27:45.244359 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:45.244346 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2: secret "agc-cred-75862bf33366" not found Oct 30 02:27:45.244520 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:45.244498 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials podName:50dd61e3-e9b2-41ed-b448-d7a97af6f1f2 nodeName:}" failed. No retries permitted until 2026-10-30 02:27:46.244478612 +0000 UTC m=+3544.747410830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" (UID: "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2") : secret "agc-cred-75862bf33366" not found Oct 30 02:27:45.300140 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.300051 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl66f\" (UniqueName: \"kubernetes.io/projected/f1fd086c-87ea-4a37-8ab3-d909e528c75a-kube-api-access-kl66f\") pod \"payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs\" (UID: \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:45.339138 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.339101 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:45.661940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:45.661900 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs"] Oct 30 02:27:45.662640 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:27:45.662615 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fd086c_87ea_4a37_8ab3_d909e528c75a.slice/crio-82d9c9f5a152b01ccab504cfeaf81c4d63cd98160c7960dd556f876837f1bddf WatchSource:0}: Error finding container 82d9c9f5a152b01ccab504cfeaf81c4d63cd98160c7960dd556f876837f1bddf: Status 404 returned error can't find the container with id 82d9c9f5a152b01ccab504cfeaf81c4d63cd98160c7960dd556f876837f1bddf Oct 30 02:27:46.260109 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:46.260007 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-000edd/agc-cred-75862bf33366: secret "agc-cred-75862bf33366" not found Oct 30 02:27:46.260109 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:46.260046 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2: secret "agc-cred-75862bf33366" not found Oct 30 02:27:46.260109 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:46.260112 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials podName:50dd61e3-e9b2-41ed-b448-d7a97af6f1f2 nodeName:}" failed. No retries permitted until 2026-10-30 02:27:48.26009165 +0000 UTC m=+3546.763023865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" (UID: "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2") : secret "agc-cred-75862bf33366" not found Oct 30 02:27:46.364181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:46.364146 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" event={"ID":"f1fd086c-87ea-4a37-8ab3-d909e528c75a","Type":"ContainerStarted","Data":"2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d"} Oct 30 02:27:46.364378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:46.364202 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" event={"ID":"f1fd086c-87ea-4a37-8ab3-d909e528c75a","Type":"ContainerStarted","Data":"82d9c9f5a152b01ccab504cfeaf81c4d63cd98160c7960dd556f876837f1bddf"} Oct 30 02:27:46.364378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:46.364253 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:46.421048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:46.420987 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" podStartSLOduration=2.420967235 podStartE2EDuration="2.420967235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:46.414890654 +0000 UTC m=+3544.917822875" watchObservedRunningTime="2026-10-30 02:27:46.420967235 +0000 UTC m=+3544.923899455" Oct 30 02:27:47.633522 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:47.633480 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:27:48.294130 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:48.294074 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-000edd/agc-cred-75862bf33366: secret "agc-cred-75862bf33366" not found Oct 30 02:27:48.294130 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:48.294125 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2: secret "agc-cred-75862bf33366" not found Oct 30 02:27:48.294413 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:48.294199 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials podName:50dd61e3-e9b2-41ed-b448-d7a97af6f1f2 nodeName:}" failed. No retries permitted until 2026-10-30 02:27:52.294178329 +0000 UTC m=+3550.797110553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" (UID: "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2") : secret "agc-cred-75862bf33366" not found Oct 30 02:27:50.308763 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:50.308723 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:27:50.393069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:50.393023 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2"] Oct 30 02:27:51.317990 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:51.317963 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:27:51.409119 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:51.409080 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52"] Oct 30 02:27:52.364649 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:52.364608 5035 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-000edd/agc-cred-75862bf33366: secret "agc-cred-75862bf33366" not found Oct 30 02:27:52.364649 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:52.364644 5035 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2: secret "agc-cred-75862bf33366" not found Oct 30 02:27:52.365175 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:52.364704 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials podName:50dd61e3-e9b2-41ed-b448-d7a97af6f1f2 nodeName:}" failed. No retries permitted until 2026-10-30 02:28:00.364689111 +0000 UTC m=+3558.867621313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" (UID: "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2") : secret "agc-cred-75862bf33366" not found Oct 30 02:27:55.394054 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.393942 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" podUID="50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" containerName="payload-processing" containerID="cri-o://fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819" gracePeriod=25 Oct 30 02:27:55.706170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.706139 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:27:55.822076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.822026 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials\") pod \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " Oct 30 02:27:55.822305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.822138 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-kube-api-access-pr9tp\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-kube-api-access-pr9tp\") pod \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " Oct 30 02:27:55.822968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.822468 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-plugins-config-volume\") pod \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " Oct 30 02:27:55.822968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.822652 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-routing-overlay-volume\") pod \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\" (UID: \"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2\") " Oct 30 02:27:55.823177 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.823024 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-plugins-config-volume" pod "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" (UID: "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:55.823177 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.823090 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-routing-overlay-volume" pod "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" (UID: "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:55.823476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.823449 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:55.823668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.823480 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:55.825211 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.825174 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-kube-api-access-pr9tp" pod "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" (UID: "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2"). InnerVolumeSpecName "kube-api-access-pr9tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:55.825381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.825356 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials" pod "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" (UID: "50dd61e3-e9b2-41ed-b448-d7a97af6f1f2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:55.925685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.925586 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:55.925685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:55.925630 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr9tp\" (UniqueName: \"kubernetes.io/projected/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2-kube-api-access-pr9tp\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:56.409862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.409818 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" podUID="a2f3940f-2d6e-4527-bb54-3c48ee35c702" containerName="payload-processing" containerID="cri-o://81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352" gracePeriod=25 Oct 30 02:27:56.471894 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.471855 5035 generic.go:356] "Generic (PLEG): container finished" podID="50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" containerID="fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819" exitCode=0 Oct 30 02:27:56.472124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.471928 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" Oct 30 02:27:56.472124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.471928 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" event={"ID":"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2","Type":"ContainerDied","Data":"fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819"} Oct 30 02:27:56.472124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.472077 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2" event={"ID":"50dd61e3-e9b2-41ed-b448-d7a97af6f1f2","Type":"ContainerDied","Data":"3b4a01b78f1b1d5a3dc84f49ff3321e88bf50dee8fc280e31f73c4bbac5b4eb2"} Oct 30 02:27:56.472124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.472109 5035 scope.go:122] "RemoveContainer" containerID="fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819" Oct 30 02:27:56.524648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.524608 5035 scope.go:122] "RemoveContainer" containerID="fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819" Oct 30 02:27:56.524975 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:56.524951 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819\": container with ID starting with fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819 not found: ID does not exist" containerID="fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819" Oct 30 02:27:56.525061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.524988 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819"} err="failed to get container status \"fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819\": rpc error: code = NotFound desc = could not find container \"fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819\": container with ID starting with fe6af315d27cfefac5fae3bb7a586882a5f275c5061d6e9d3a14a21edb786819 not found: ID does not exist" Oct 30 02:27:56.543556 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.543505 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2"] Oct 30 02:27:56.559290 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.559251 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-58cdqqtk2"] Oct 30 02:27:56.710649 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.710618 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:27:56.837810 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.837764 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-kube-api-access-2jmlt\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-kube-api-access-2jmlt\") pod \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " Oct 30 02:27:56.838034 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.837819 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-routing-overlay-volume\") pod \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " Oct 30 02:27:56.838215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.838188 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-routing-overlay-volume" pod "a2f3940f-2d6e-4527-bb54-3c48ee35c702" (UID: "a2f3940f-2d6e-4527-bb54-3c48ee35c702"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:56.838282 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.838223 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-provider-credentials\") pod \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " Oct 30 02:27:56.838351 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.838309 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-plugins-config-volume\") pod \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\" (UID: \"a2f3940f-2d6e-4527-bb54-3c48ee35c702\") " Oct 30 02:27:56.838748 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.838718 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-plugins-config-volume" pod "a2f3940f-2d6e-4527-bb54-3c48ee35c702" (UID: "a2f3940f-2d6e-4527-bb54-3c48ee35c702"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:56.839000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.838980 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:56.839078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.839006 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2f3940f-2d6e-4527-bb54-3c48ee35c702-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:56.840464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.840441 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-kube-api-access-2jmlt" pod "a2f3940f-2d6e-4527-bb54-3c48ee35c702" (UID: "a2f3940f-2d6e-4527-bb54-3c48ee35c702"). InnerVolumeSpecName "kube-api-access-2jmlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:56.840606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.840471 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-provider-credentials" pod "a2f3940f-2d6e-4527-bb54-3c48ee35c702" (UID: "a2f3940f-2d6e-4527-bb54-3c48ee35c702"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:56.940465 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.940374 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:56.940465 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:56.940410 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jmlt\" (UniqueName: \"kubernetes.io/projected/a2f3940f-2d6e-4527-bb54-3c48ee35c702-kube-api-access-2jmlt\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:27:57.376923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.376891 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:27:57.486280 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.486237 5035 generic.go:356] "Generic (PLEG): container finished" podID="a2f3940f-2d6e-4527-bb54-3c48ee35c702" containerID="81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352" exitCode=0 Oct 30 02:27:57.486863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.486299 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" Oct 30 02:27:57.486863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.486411 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" event={"ID":"a2f3940f-2d6e-4527-bb54-3c48ee35c702","Type":"ContainerDied","Data":"81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352"} Oct 30 02:27:57.486863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.486460 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52" event={"ID":"a2f3940f-2d6e-4527-bb54-3c48ee35c702","Type":"ContainerDied","Data":"f56ab57bb935538d2fec7d2e45ec18806350545c400ef3f2362c99bfe1651dfe"} Oct 30 02:27:57.486863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.486494 5035 scope.go:122] "RemoveContainer" containerID="81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352" Oct 30 02:27:57.502920 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.502790 5035 scope.go:122] "RemoveContainer" containerID="81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352" Oct 30 02:27:57.503738 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:27:57.503663 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352\": container with ID starting with 81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352 not found: ID does not exist" containerID="81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352" Oct 30 02:27:57.503738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.503703 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352"} err="failed to get container status \"81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352\": rpc error: code = NotFound desc = could not find container \"81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352\": container with ID starting with 81354b909729187d5f948322a818a66a99806c32d8ab5cbbd83357f27a129352 not found: ID does not exist" Oct 30 02:27:57.598481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.598382 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52"] Oct 30 02:27:57.613696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.613615 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-6fb822m52"] Oct 30 02:27:57.646220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.646134 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" path="/var/lib/kubelet/pods/50dd61e3-e9b2-41ed-b448-d7a97af6f1f2/volumes" Oct 30 02:27:57.646744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:27:57.646726 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f3940f-2d6e-4527-bb54-3c48ee35c702" path="/var/lib/kubelet/pods/a2f3940f-2d6e-4527-bb54-3c48ee35c702/volumes" Oct 30 02:28:00.450959 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.450919 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd"] Oct 30 02:28:00.452130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.452102 5035 cpu_manager.go:414] "containerMap: removing container" podUID="a2f3940f-2d6e-4527-bb54-3c48ee35c702" containerName="payload-processing" Oct 30 02:28:00.452130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.452130 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f3940f-2d6e-4527-bb54-3c48ee35c702" containerName="payload-processing" Oct 30 02:28:00.452337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.452173 5035 cpu_manager.go:414] "containerMap: removing container" podUID="50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" containerName="payload-processing" Oct 30 02:28:00.452337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.452181 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" containerName="payload-processing" Oct 30 02:28:00.452514 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.452498 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f3940f-2d6e-4527-bb54-3c48ee35c702" containerName="payload-processing" Oct 30 02:28:00.452601 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.452540 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="50dd61e3-e9b2-41ed-b448-d7a97af6f1f2" containerName="payload-processing" Oct 30 02:28:00.456926 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.456899 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.474066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.474025 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4593af-serving-cert\"" Oct 30 02:28:00.479241 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.479204 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd"] Oct 30 02:28:00.630456 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.630385 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d492b6e-b398-48f6-b279-ef1ccd93f472-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.630862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.630745 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n495\" (UniqueName: \"kubernetes.io/projected/7d492b6e-b398-48f6-b279-ef1ccd93f472-kube-api-access-8n495\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.630958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.630934 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-metrics-certs\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.631027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.631010 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-maas-api-tls\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.734481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.734371 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n495\" (UniqueName: \"kubernetes.io/projected/7d492b6e-b398-48f6-b279-ef1ccd93f472-kube-api-access-8n495\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.735472 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.735433 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-metrics-certs\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.737150 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.737114 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-maas-api-tls\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.737962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.737922 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d492b6e-b398-48f6-b279-ef1ccd93f472-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.739504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.739468 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d492b6e-b398-48f6-b279-ef1ccd93f472-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.742182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.742152 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-maas-api-tls\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.742635 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.742605 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-metrics-certs\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.761171 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.761133 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n495\" (UniqueName: \"kubernetes.io/projected/7d492b6e-b398-48f6-b279-ef1ccd93f472-kube-api-access-8n495\") pod \"maas-api-e2e-inf-b-4593af-5989484456-chspd\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:00.778264 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:00.778218 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:01.137590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:01.137176 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd"] Oct 30 02:28:01.285177 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:01.285133 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v"] Oct 30 02:28:01.527868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:01.527819 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" event={"ID":"7d492b6e-b398-48f6-b279-ef1ccd93f472","Type":"ContainerStarted","Data":"f047a44bb33571d19cbb5981e934498c326a8f1adae4d34e2c18041fa959f2f1"} Oct 30 02:28:02.263521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:02.263482 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt"] Oct 30 02:28:02.538911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:02.538808 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" event={"ID":"7d492b6e-b398-48f6-b279-ef1ccd93f472","Type":"ContainerStarted","Data":"ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7"} Oct 30 02:28:02.539409 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:02.538916 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:02.591306 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:02.591238 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" podStartSLOduration=2.279092458 podStartE2EDuration="2.591217558s" totalImagesPullingTime="312.1251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:28:02.589735175 +0000 UTC m=+3561.092667397" watchObservedRunningTime="2026-10-30 02:28:02.591217558 +0000 UTC m=+3561.094149779" Oct 30 02:28:06.286132 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.286074 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" podUID="26f65130-417f-46b7-97ef-a68f9daf137a" containerName="payload-processing" containerID="cri-o://d71b9462e8bb4b99b6da72f6b827e4fd34da21daaa44e135116df63174c66423" gracePeriod=25 Oct 30 02:28:06.590825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.590786 5035 generic.go:356] "Generic (PLEG): container finished" podID="26f65130-417f-46b7-97ef-a68f9daf137a" containerID="d71b9462e8bb4b99b6da72f6b827e4fd34da21daaa44e135116df63174c66423" exitCode=0 Oct 30 02:28:06.591005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.590860 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" event={"ID":"26f65130-417f-46b7-97ef-a68f9daf137a","Type":"ContainerDied","Data":"d71b9462e8bb4b99b6da72f6b827e4fd34da21daaa44e135116df63174c66423"} Oct 30 02:28:06.591005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.590934 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" event={"ID":"26f65130-417f-46b7-97ef-a68f9daf137a","Type":"ContainerDied","Data":"0d8d99f3b2dbe66f5c3c1f359041e4109334e77750e3e7082350f98f1665ee30"} Oct 30 02:28:06.591005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.590957 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8d99f3b2dbe66f5c3c1f359041e4109334e77750e3e7082350f98f1665ee30" Oct 30 02:28:06.611514 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.611484 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:28:06.756654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.756613 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-provider-credentials\") pod \"26f65130-417f-46b7-97ef-a68f9daf137a\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " Oct 30 02:28:06.756878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.756829 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-kube-api-access-th9b6\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-kube-api-access-th9b6\") pod \"26f65130-417f-46b7-97ef-a68f9daf137a\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " Oct 30 02:28:06.756954 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.756882 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-routing-overlay-volume\") pod \"26f65130-417f-46b7-97ef-a68f9daf137a\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " Oct 30 02:28:06.757014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.756997 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-plugins-config-volume\") pod \"26f65130-417f-46b7-97ef-a68f9daf137a\" (UID: \"26f65130-417f-46b7-97ef-a68f9daf137a\") " Oct 30 02:28:06.757298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.757269 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-routing-overlay-volume" pod "26f65130-417f-46b7-97ef-a68f9daf137a" (UID: "26f65130-417f-46b7-97ef-a68f9daf137a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:28:06.757451 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.757427 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-plugins-config-volume" pod "26f65130-417f-46b7-97ef-a68f9daf137a" (UID: "26f65130-417f-46b7-97ef-a68f9daf137a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:28:06.758068 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.758039 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:28:06.758167 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.758073 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f65130-417f-46b7-97ef-a68f9daf137a-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:28:06.759195 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.759170 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-provider-credentials" pod "26f65130-417f-46b7-97ef-a68f9daf137a" (UID: "26f65130-417f-46b7-97ef-a68f9daf137a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:28:06.759287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.759199 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-kube-api-access-th9b6" pod "26f65130-417f-46b7-97ef-a68f9daf137a" (UID: "26f65130-417f-46b7-97ef-a68f9daf137a"). InnerVolumeSpecName "kube-api-access-th9b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:28:06.859948 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.859842 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:28:06.860144 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:06.860128 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th9b6\" (UniqueName: \"kubernetes.io/projected/26f65130-417f-46b7-97ef-a68f9daf137a-kube-api-access-th9b6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:28:07.264401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.264354 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" podUID="b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" containerName="payload-processing" containerID="cri-o://b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0" gracePeriod=25 Oct 30 02:28:07.585882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.585844 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:28:07.600867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.600824 5035 generic.go:356] "Generic (PLEG): container finished" podID="b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" containerID="b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0" exitCode=0 Oct 30 02:28:07.601052 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.600870 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" Oct 30 02:28:07.601052 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.600907 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" event={"ID":"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a","Type":"ContainerDied","Data":"b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0"} Oct 30 02:28:07.601052 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.600966 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt" event={"ID":"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a","Type":"ContainerDied","Data":"77298d8c3d51432a0cf13e006f735a0b694c5aaf0ba4ab1b9c662a45651e5847"} Oct 30 02:28:07.601052 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.600993 5035 scope.go:122] "RemoveContainer" containerID="b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0" Oct 30 02:28:07.601236 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.601093 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v" Oct 30 02:28:07.612891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.612869 5035 scope.go:122] "RemoveContainer" containerID="b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0" Oct 30 02:28:07.613222 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:28:07.613191 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0\": container with ID starting with b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0 not found: ID does not exist" containerID="b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0" Oct 30 02:28:07.613340 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.613228 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0"} err="failed to get container status \"b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0\": rpc error: code = NotFound desc = could not find container \"b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0\": container with ID starting with b3767c6bc23c8967d40f86c8143e75eed1e5fdb603e205c53685807aa2483df0 not found: ID does not exist" Oct 30 02:28:07.674935 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.674889 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-kube-api-access-9kqlg\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-kube-api-access-9kqlg\") pod \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " Oct 30 02:28:07.675149 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.675104 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-plugins-config-volume\") pod \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " Oct 30 02:28:07.675228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.675201 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-provider-credentials\") pod \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " Oct 30 02:28:07.675349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.675323 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-routing-overlay-volume\") pod \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\" (UID: \"b0a53aa1-fbf4-4e68-9605-7ba41c073f0a\") " Oct 30 02:28:07.675663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.675634 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-plugins-config-volume" pod "b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" (UID: "b0a53aa1-fbf4-4e68-9605-7ba41c073f0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:28:07.675799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.675724 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-routing-overlay-volume" pod "b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" (UID: "b0a53aa1-fbf4-4e68-9605-7ba41c073f0a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:28:07.676424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.676394 5035 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-routing-overlay-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:28:07.676558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.676429 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:28:07.677662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.677637 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-kube-api-access-9kqlg" pod "b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" (UID: "b0a53aa1-fbf4-4e68-9605-7ba41c073f0a"). InnerVolumeSpecName "kube-api-access-9kqlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:28:07.678080 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.678059 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-provider-credentials" pod "b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" (UID: "b0a53aa1-fbf4-4e68-9605-7ba41c073f0a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:28:07.694985 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.694944 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v"] Oct 30 02:28:07.700847 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.700811 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d64q4p4v"] Oct 30 02:28:07.778879 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.778845 5035 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-provider-credentials\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:28:07.778879 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.778875 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqlg\" (UniqueName: \"kubernetes.io/projected/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a-kube-api-access-9kqlg\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:28:07.954344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.954297 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt"] Oct 30 02:28:07.960755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:07.960711 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-7c4d9d5nt"] Oct 30 02:28:08.554624 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:08.554590 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:28:09.666477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:09.666433 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f65130-417f-46b7-97ef-a68f9daf137a" path="/var/lib/kubelet/pods/26f65130-417f-46b7-97ef-a68f9daf137a/volumes" Oct 30 02:28:09.667226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:09.667201 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" path="/var/lib/kubelet/pods/b0a53aa1-fbf4-4e68-9605-7ba41c073f0a/volumes" Oct 30 02:28:16.980939 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:16.980895 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw"] Oct 30 02:28:16.981862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:16.981839 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" containerName="payload-processing" Oct 30 02:28:16.981862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:16.981860 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" containerName="payload-processing" Oct 30 02:28:16.982010 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:16.981887 5035 cpu_manager.go:414] "containerMap: removing container" podUID="26f65130-417f-46b7-97ef-a68f9daf137a" containerName="payload-processing" Oct 30 02:28:16.982010 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:16.981894 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f65130-417f-46b7-97ef-a68f9daf137a" containerName="payload-processing" Oct 30 02:28:16.982117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:16.982098 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a53aa1-fbf4-4e68-9605-7ba41c073f0a" containerName="payload-processing" Oct 30 02:28:16.982117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:16.982115 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f65130-417f-46b7-97ef-a68f9daf137a" containerName="payload-processing" Oct 30 02:28:16.986202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:16.986179 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.009199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.009165 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw"] Oct 30 02:28:17.011209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.011180 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c75203-serving-cert\"" Oct 30 02:28:17.034778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.034719 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca538fb-fd56-45cb-9d60-dc952c70d5ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.034997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.034906 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.034997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.034949 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-metrics-certs\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.035127 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.035017 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfwr6\" (UniqueName: \"kubernetes.io/projected/dca538fb-fd56-45cb-9d60-dc952c70d5ba-kube-api-access-mfwr6\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.136349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.136310 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca538fb-fd56-45cb-9d60-dc952c70d5ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.136582 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.136494 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.136582 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.136567 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-metrics-certs\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.136717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.136617 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfwr6\" (UniqueName: \"kubernetes.io/projected/dca538fb-fd56-45cb-9d60-dc952c70d5ba-kube-api-access-mfwr6\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.137171 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.137146 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca538fb-fd56-45cb-9d60-dc952c70d5ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.139201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.139177 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-metrics-certs\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.139201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.139189 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.163054 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.163021 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfwr6\" (UniqueName: \"kubernetes.io/projected/dca538fb-fd56-45cb-9d60-dc952c70d5ba-kube-api-access-mfwr6\") pod \"maas-api-e2e-inf-b-c75203-7d5f897748-r27fw\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.304746 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.304643 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:17.633187 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:28:17.633099 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:28:17.636922 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:28:17.636890 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca538fb_fd56_45cb_9d60_dc952c70d5ba.slice/crio-d4fe478ce26598f150a398e28550d3bdddf8fd2ded539c9af754c4fff4781aa0 WatchSource:0}: Error finding container d4fe478ce26598f150a398e28550d3bdddf8fd2ded539c9af754c4fff4781aa0: Status 404 returned error can't find the container with id d4fe478ce26598f150a398e28550d3bdddf8fd2ded539c9af754c4fff4781aa0 Oct 30 02:28:17.646710 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.646680 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw"] Oct 30 02:28:17.688377 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:17.688340 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" event={"ID":"dca538fb-fd56-45cb-9d60-dc952c70d5ba","Type":"ContainerStarted","Data":"d4fe478ce26598f150a398e28550d3bdddf8fd2ded539c9af754c4fff4781aa0"} Oct 30 02:28:18.713000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:18.712957 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" event={"ID":"dca538fb-fd56-45cb-9d60-dc952c70d5ba","Type":"ContainerStarted","Data":"71614811bf2de8a20618bff6161496c974ae8b8050cfe97335200e181296f8d4"} Oct 30 02:28:18.713563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:18.713114 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:18.773229 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:18.773171 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" podStartSLOduration=2.328608524 podStartE2EDuration="2.773156813s" totalImagesPullingTime="444.548289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:28:18.773088009 +0000 UTC m=+3577.276020223" watchObservedRunningTime="2026-10-30 02:28:18.773156813 +0000 UTC m=+3577.276089033" Oct 30 02:28:23.686079 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:23.686034 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t"] Oct 30 02:28:23.693321 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:23.693294 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:23.712598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:23.712559 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t"] Oct 30 02:28:23.719818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:23.719767 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-52d131\"" Oct 30 02:28:23.720039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:23.719928 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-52d131-dockercfg-wg65b\"" Oct 30 02:28:24.061645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:24.061596 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrrlm\" (UniqueName: \"kubernetes.io/projected/0b3d0792-204a-4487-9ea1-efb6b6616c70-kube-api-access-jrrlm\") pod \"payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t\" (UID: \"0b3d0792-204a-4487-9ea1-efb6b6616c70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:24.061872 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:24.061827 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3d0792-204a-4487-9ea1-efb6b6616c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t\" (UID: \"0b3d0792-204a-4487-9ea1-efb6b6616c70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:24.163203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:24.163153 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3d0792-204a-4487-9ea1-efb6b6616c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t\" (UID: \"0b3d0792-204a-4487-9ea1-efb6b6616c70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:24.163461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:24.163444 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrlm\" (UniqueName: \"kubernetes.io/projected/0b3d0792-204a-4487-9ea1-efb6b6616c70-kube-api-access-jrrlm\") pod \"payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t\" (UID: \"0b3d0792-204a-4487-9ea1-efb6b6616c70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:24.164074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:24.164050 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3d0792-204a-4487-9ea1-efb6b6616c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t\" (UID: \"0b3d0792-204a-4487-9ea1-efb6b6616c70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:24.192278 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:24.192232 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrrlm\" (UniqueName: \"kubernetes.io/projected/0b3d0792-204a-4487-9ea1-efb6b6616c70-kube-api-access-jrrlm\") pod \"payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t\" (UID: \"0b3d0792-204a-4487-9ea1-efb6b6616c70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:24.310646 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:24.310604 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:24.633540 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:24.633498 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t"] Oct 30 02:28:24.633926 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:28:24.633900 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3d0792_204a_4487_9ea1_efb6b6616c70.slice/crio-0304bde404351a224b08893365e1b09e2c849a9c02c930c839aa60b46149a7ad WatchSource:0}: Error finding container 0304bde404351a224b08893365e1b09e2c849a9c02c930c839aa60b46149a7ad: Status 404 returned error can't find the container with id 0304bde404351a224b08893365e1b09e2c849a9c02c930c839aa60b46149a7ad Oct 30 02:28:24.728385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:24.728352 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:28:25.002193 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:25.002156 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" event={"ID":"0b3d0792-204a-4487-9ea1-efb6b6616c70","Type":"ContainerStarted","Data":"f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602"} Oct 30 02:28:25.002404 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:25.002212 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" event={"ID":"0b3d0792-204a-4487-9ea1-efb6b6616c70","Type":"ContainerStarted","Data":"0304bde404351a224b08893365e1b09e2c849a9c02c930c839aa60b46149a7ad"} Oct 30 02:28:25.002404 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:25.002244 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:25.051627 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:25.051556 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" podStartSLOduration=2.051518694 podStartE2EDuration="2.051518694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:28:25.046220755 +0000 UTC m=+3583.549153181" watchObservedRunningTime="2026-10-30 02:28:25.051518694 +0000 UTC m=+3583.554450915" Oct 30 02:28:36.014478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:36.014447 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:28:43.947782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:43.947752 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:28:43.964370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:43.964337 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:28:44.012965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:44.012934 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:28:44.030067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:44.030036 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:28:47.218576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:47.218540 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:28:47.219026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:47.218597 5035 generic.go:356] "Generic (PLEG): container finished" podID="bd24e16c-c31b-4af0-b518-cfe8c85937ac" containerID="33fa605b4bf6d7dd7871e062bc830161f1354da3be2d738c75fd9ee076ac0819" exitCode=2 Oct 30 02:28:47.219026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:47.218681 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" event={"ID":"bd24e16c-c31b-4af0-b518-cfe8c85937ac","Type":"ContainerDied","Data":"33fa605b4bf6d7dd7871e062bc830161f1354da3be2d738c75fd9ee076ac0819"} Oct 30 02:28:47.219392 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:47.219370 5035 scope.go:122] "RemoveContainer" containerID="33fa605b4bf6d7dd7871e062bc830161f1354da3be2d738c75fd9ee076ac0819" Oct 30 02:28:47.632823 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:28:47.632732 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:28:48.230567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:48.230515 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:28:48.231028 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:28:48.230613 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5kdp7" event={"ID":"bd24e16c-c31b-4af0-b518-cfe8c85937ac","Type":"ContainerStarted","Data":"d4faa50752f238c781125b7feabb26891e60b9facffa04f23b422ce894be2236"} Oct 30 02:29:16.601148 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:16.601089 5035 generic.go:356] "Generic (PLEG): container finished" podID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerID="e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3" exitCode=0 Oct 30 02:29:16.602225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:16.601810 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" event={"ID":"d27674de-a2e2-4e4f-82a2-00e44a555270","Type":"ContainerDied","Data":"e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3"} Oct 30 02:29:16.603246 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:16.603144 5035 scope.go:122] "RemoveContainer" containerID="e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3" Oct 30 02:29:16.610050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:16.610003 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:29:16.610173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:16.610123 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:29:16.610235 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:16.610174 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:29:17.615459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:17.615420 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" event={"ID":"d27674de-a2e2-4e4f-82a2-00e44a555270","Type":"ContainerStarted","Data":"555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272"} Oct 30 02:29:17.633722 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:29:17.633683 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:29:18.305022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.304259 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt"] Oct 30 02:29:18.323106 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.323069 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt"] Oct 30 02:29:18.323384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.323192 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.328480 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.328402 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:29:18.330983 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.329456 5035 patch_prober.go:29] interesting pod/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:29:18.330983 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.329509 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 02:29:18.341688 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.341663 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-52d131-serving-cert\"" Oct 30 02:29:18.418473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.418425 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98d14cd-45be-457a-a181-3a96931d419e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.418473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.418482 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-metrics-certs\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.418780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.418654 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2pfm\" (UniqueName: \"kubernetes.io/projected/e98d14cd-45be-457a-a181-3a96931d419e-kube-api-access-b2pfm\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.418780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.418716 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-maas-api-tls\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.521904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.521854 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98d14cd-45be-457a-a181-3a96931d419e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.523150 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.523121 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-metrics-certs\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.524121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.524076 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pfm\" (UniqueName: \"kubernetes.io/projected/e98d14cd-45be-457a-a181-3a96931d419e-kube-api-access-b2pfm\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.524488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.524444 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-maas-api-tls\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.525453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.523003 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98d14cd-45be-457a-a181-3a96931d419e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.541797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.541594 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-maas-api-tls\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.562741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.557539 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pfm\" (UniqueName: \"kubernetes.io/projected/e98d14cd-45be-457a-a181-3a96931d419e-kube-api-access-b2pfm\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.562741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.560423 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-metrics-certs\") pod \"maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.646478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.646442 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:18.999809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:18.999768 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt"] Oct 30 02:29:19.005674 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:29:19.005630 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98d14cd_45be_457a_a181_3a96931d419e.slice/crio-bb1c3a50b5f25c73b99061968762d7117bcb6baf2196e98e1e7f22a535056f2f WatchSource:0}: Error finding container bb1c3a50b5f25c73b99061968762d7117bcb6baf2196e98e1e7f22a535056f2f: Status 404 returned error can't find the container with id bb1c3a50b5f25c73b99061968762d7117bcb6baf2196e98e1e7f22a535056f2f Oct 30 02:29:19.326738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:19.326645 5035 patch_prober.go:29] interesting pod/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:29:19.326738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:19.326702 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 02:29:19.651488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:19.651452 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" event={"ID":"e98d14cd-45be-457a-a181-3a96931d419e","Type":"ContainerStarted","Data":"bb1c3a50b5f25c73b99061968762d7117bcb6baf2196e98e1e7f22a535056f2f"} Oct 30 02:29:20.327477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:20.327434 5035 patch_prober.go:29] interesting pod/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:29:20.327700 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:20.327495 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 02:29:20.654257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:20.654142 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" event={"ID":"e98d14cd-45be-457a-a181-3a96931d419e","Type":"ContainerStarted","Data":"5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb"} Oct 30 02:29:20.654257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:20.654248 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:20.719674 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:20.719616 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" podStartSLOduration=2.268926913 podStartE2EDuration="2.719597431s" totalImagesPullingTime="450.670518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:29:20.71916223 +0000 UTC m=+3639.222094462" watchObservedRunningTime="2026-10-30 02:29:20.719597431 +0000 UTC m=+3639.222529650" Oct 30 02:29:21.097856 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:21.097807 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:29:21.326650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:21.326591 5035 patch_prober.go:29] interesting pod/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:29:21.326858 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:21.326683 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 02:29:22.327381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:22.327338 5035 patch_prober.go:29] interesting pod/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:29:22.327868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:22.327403 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 30 02:29:23.333943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:23.333859 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:29:23.335505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:23.335433 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:29:26.671172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:29:26.671139 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:29:47.631932 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:29:47.631880 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:30:00.256875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.256829 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r"] Oct 30 02:30:00.264807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.264715 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:00.288458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.288422 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r"] Oct 30 02:30:00.295017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.294985 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm"] Oct 30 02:30:00.338830 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.333762 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x"] Oct 30 02:30:00.339510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.339473 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:00.341049 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.340994 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm"] Oct 30 02:30:00.342142 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.341113 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:00.343950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.343922 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7"] Oct 30 02:30:00.351362 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.351329 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:00.352714 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.352681 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdgvj\" (UniqueName: \"kubernetes.io/projected/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-kube-api-access-tdgvj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r\" (UID: \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:00.352971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.352951 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r\" (UID: \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:00.361875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.361824 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x"] Oct 30 02:30:00.366977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.366819 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s"] Oct 30 02:30:00.380659 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.380627 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt"] Oct 30 02:30:00.383208 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.382023 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:00.383716 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.383458 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:30:00.390179 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.390142 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7"] Oct 30 02:30:00.390350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.390272 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:00.393869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.393830 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s"] Oct 30 02:30:00.405639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.405595 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt"] Oct 30 02:30:00.454574 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.454517 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2a4698-c3d7-4af1-9903-76566281a8ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7\" (UID: \"bf2a4698-c3d7-4af1-9903-76566281a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:00.454776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.454642 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt\" (UID: \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:00.454776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.454681 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlz9z\" (UniqueName: \"kubernetes.io/projected/bf2a4698-c3d7-4af1-9903-76566281a8ec-kube-api-access-jlz9z\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7\" (UID: \"bf2a4698-c3d7-4af1-9903-76566281a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:00.454776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.454736 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdgvj\" (UniqueName: \"kubernetes.io/projected/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-kube-api-access-tdgvj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r\" (UID: \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:00.454896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.454855 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzw6\" (UniqueName: \"kubernetes.io/projected/103ec715-4f09-41c8-adba-c72457e4e9e1-kube-api-access-7lzw6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s\" (UID: \"103ec715-4f09-41c8-adba-c72457e4e9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:00.454954 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.454936 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x\" (UID: \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:00.455009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.454979 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103ec715-4f09-41c8-adba-c72457e4e9e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s\" (UID: \"103ec715-4f09-41c8-adba-c72457e4e9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:00.455059 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.455037 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4krt\" (UniqueName: \"kubernetes.io/projected/e8bb030d-d1b0-4485-8d30-3fbdfa293226-kube-api-access-n4krt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm\" (UID: \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:00.455147 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.455131 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bb030d-d1b0-4485-8d30-3fbdfa293226-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm\" (UID: \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:00.455185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.455162 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2b2q\" (UniqueName: \"kubernetes.io/projected/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-kube-api-access-m2b2q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt\" (UID: \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:00.455225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.455212 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r\" (UID: \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:00.455317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.455306 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwfj\" (UniqueName: \"kubernetes.io/projected/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-kube-api-access-jtwfj\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x\" (UID: \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:00.455852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.455833 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r\" (UID: \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:00.483127 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.483088 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdgvj\" (UniqueName: \"kubernetes.io/projected/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-kube-api-access-tdgvj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r\" (UID: \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:00.557416 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.557316 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bb030d-d1b0-4485-8d30-3fbdfa293226-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm\" (UID: \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:00.557416 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.557368 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2b2q\" (UniqueName: \"kubernetes.io/projected/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-kube-api-access-m2b2q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt\" (UID: \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:00.557771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.557422 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwfj\" (UniqueName: \"kubernetes.io/projected/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-kube-api-access-jtwfj\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x\" (UID: \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:00.557771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.557574 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2a4698-c3d7-4af1-9903-76566281a8ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7\" (UID: \"bf2a4698-c3d7-4af1-9903-76566281a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:00.557771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.557644 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt\" (UID: \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:00.557771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.557728 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlz9z\" (UniqueName: \"kubernetes.io/projected/bf2a4698-c3d7-4af1-9903-76566281a8ec-kube-api-access-jlz9z\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7\" (UID: \"bf2a4698-c3d7-4af1-9903-76566281a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:00.558013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.557901 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzw6\" (UniqueName: \"kubernetes.io/projected/103ec715-4f09-41c8-adba-c72457e4e9e1-kube-api-access-7lzw6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s\" (UID: \"103ec715-4f09-41c8-adba-c72457e4e9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:00.558013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.557993 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x\" (UID: \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:00.558128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.558058 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103ec715-4f09-41c8-adba-c72457e4e9e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s\" (UID: \"103ec715-4f09-41c8-adba-c72457e4e9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:00.558128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.558088 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4krt\" (UniqueName: \"kubernetes.io/projected/e8bb030d-d1b0-4485-8d30-3fbdfa293226-kube-api-access-n4krt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm\" (UID: \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:00.558548 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.558286 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bb030d-d1b0-4485-8d30-3fbdfa293226-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm\" (UID: \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:00.558548 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.558379 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt\" (UID: \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:00.558548 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.558450 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2a4698-c3d7-4af1-9903-76566281a8ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7\" (UID: \"bf2a4698-c3d7-4af1-9903-76566281a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:00.558927 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.558903 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x\" (UID: \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:00.559030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.559003 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103ec715-4f09-41c8-adba-c72457e4e9e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s\" (UID: \"103ec715-4f09-41c8-adba-c72457e4e9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:00.583375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.583336 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2b2q\" (UniqueName: \"kubernetes.io/projected/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-kube-api-access-m2b2q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt\" (UID: \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:00.583656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.583635 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwfj\" (UniqueName: \"kubernetes.io/projected/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-kube-api-access-jtwfj\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x\" (UID: \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:00.583737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.583650 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4krt\" (UniqueName: \"kubernetes.io/projected/e8bb030d-d1b0-4485-8d30-3fbdfa293226-kube-api-access-n4krt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm\" (UID: \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:00.583737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.583675 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlz9z\" (UniqueName: \"kubernetes.io/projected/bf2a4698-c3d7-4af1-9903-76566281a8ec-kube-api-access-jlz9z\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7\" (UID: \"bf2a4698-c3d7-4af1-9903-76566281a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:00.583817 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.583765 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzw6\" (UniqueName: \"kubernetes.io/projected/103ec715-4f09-41c8-adba-c72457e4e9e1-kube-api-access-7lzw6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s\" (UID: \"103ec715-4f09-41c8-adba-c72457e4e9e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:00.629879 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.629840 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:00.687050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.687004 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:00.703064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.702914 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:00.715387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.715156 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:00.728475 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.728340 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:00.743810 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:00.743732 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:01.206497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:01.206438 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r"] Oct 30 02:30:01.538105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:01.538071 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7"] Oct 30 02:30:01.552372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:01.552338 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s"] Oct 30 02:30:01.557834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:01.557803 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt"] Oct 30 02:30:01.571343 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:30:01.571299 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2a4698_c3d7_4af1_9903_76566281a8ec.slice/crio-c75ca8ffdd8362a67b7e0ff3f427ceacc3c727378d26db5b7a7cf7de06bedab2 WatchSource:0}: Error finding container c75ca8ffdd8362a67b7e0ff3f427ceacc3c727378d26db5b7a7cf7de06bedab2: Status 404 returned error can't find the container with id c75ca8ffdd8362a67b7e0ff3f427ceacc3c727378d26db5b7a7cf7de06bedab2 Oct 30 02:30:01.572627 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:30:01.572591 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103ec715_4f09_41c8_adba_c72457e4e9e1.slice/crio-3ce3cc3817464a2b4960aff1d68382420ba77867c4783aa8f1d5183f0b95a9c6 WatchSource:0}: Error finding container 3ce3cc3817464a2b4960aff1d68382420ba77867c4783aa8f1d5183f0b95a9c6: Status 404 returned error can't find the container with id 3ce3cc3817464a2b4960aff1d68382420ba77867c4783aa8f1d5183f0b95a9c6 Oct 30 02:30:01.574630 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:30:01.574607 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88bbf10_8b58_4b5d_a7f1_2ea77c6c18da.slice/crio-8db2b91b69af58e4afb8ba2b11150785ca21ad8436276bf7d97f6f3e2c65a282 WatchSource:0}: Error finding container 8db2b91b69af58e4afb8ba2b11150785ca21ad8436276bf7d97f6f3e2c65a282: Status 404 returned error can't find the container with id 8db2b91b69af58e4afb8ba2b11150785ca21ad8436276bf7d97f6f3e2c65a282 Oct 30 02:30:01.590678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:01.590612 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x"] Oct 30 02:30:01.595833 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:01.595754 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm"] Oct 30 02:30:01.598844 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:30:01.598807 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bb030d_d1b0_4485_8d30_3fbdfa293226.slice/crio-1f82b8707ec5cff3edd2fae0eada569be27ed03feb94957dd6555f0949c03f27 WatchSource:0}: Error finding container 1f82b8707ec5cff3edd2fae0eada569be27ed03feb94957dd6555f0949c03f27: Status 404 returned error can't find the container with id 1f82b8707ec5cff3edd2fae0eada569be27ed03feb94957dd6555f0949c03f27 Oct 30 02:30:01.599772 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:30:01.599657 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540dd1e7_ebaf_4d58_83a1_fc2334bbfa12.slice/crio-e2f9bf194b6e1341371cdd24859d4c291c66ba9c361b766b59935a1f30417271 WatchSource:0}: Error finding container e2f9bf194b6e1341371cdd24859d4c291c66ba9c361b766b59935a1f30417271: Status 404 returned error can't find the container with id e2f9bf194b6e1341371cdd24859d4c291c66ba9c361b766b59935a1f30417271 Oct 30 02:30:02.305941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:02.305842 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" event={"ID":"103ec715-4f09-41c8-adba-c72457e4e9e1","Type":"ContainerStarted","Data":"3ce3cc3817464a2b4960aff1d68382420ba77867c4783aa8f1d5183f0b95a9c6"} Oct 30 02:30:02.340861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:02.318459 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" event={"ID":"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12","Type":"ContainerStarted","Data":"e2f9bf194b6e1341371cdd24859d4c291c66ba9c361b766b59935a1f30417271"} Oct 30 02:30:02.340861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:02.318549 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" event={"ID":"bf2a4698-c3d7-4af1-9903-76566281a8ec","Type":"ContainerStarted","Data":"c75ca8ffdd8362a67b7e0ff3f427ceacc3c727378d26db5b7a7cf7de06bedab2"} Oct 30 02:30:02.340861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:02.318579 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" event={"ID":"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da","Type":"ContainerStarted","Data":"8db2b91b69af58e4afb8ba2b11150785ca21ad8436276bf7d97f6f3e2c65a282"} Oct 30 02:30:02.340861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:02.318602 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" event={"ID":"e8bb030d-d1b0-4485-8d30-3fbdfa293226","Type":"ContainerStarted","Data":"1f82b8707ec5cff3edd2fae0eada569be27ed03feb94957dd6555f0949c03f27"} Oct 30 02:30:02.340861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:02.318625 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" event={"ID":"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4","Type":"ContainerStarted","Data":"e4b88a26ced8b94c9f2552c61f85611b91712ef5192d1ad16f9338e52f673574"} Oct 30 02:30:03.329438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.329397 5035 generic.go:356] "Generic (PLEG): container finished" podID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerID="91031bcb2a3dc531b622488cc1de01fd40fbdfce71073cb7ab007c442af5f751" exitCode=60 Oct 30 02:30:03.329989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.329488 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" event={"ID":"bf2a4698-c3d7-4af1-9903-76566281a8ec","Type":"ContainerDied","Data":"91031bcb2a3dc531b622488cc1de01fd40fbdfce71073cb7ab007c442af5f751"} Oct 30 02:30:03.329989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.329931 5035 scope.go:122] "RemoveContainer" containerID="91031bcb2a3dc531b622488cc1de01fd40fbdfce71073cb7ab007c442af5f751" Oct 30 02:30:03.331501 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.331476 5035 generic.go:356] "Generic (PLEG): container finished" podID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerID="cd06c2e14767e6dd6cd067a9577ecd93552407a3d05f85b955612bd6b29eb0f2" exitCode=60 Oct 30 02:30:03.331656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.331599 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" event={"ID":"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da","Type":"ContainerDied","Data":"cd06c2e14767e6dd6cd067a9577ecd93552407a3d05f85b955612bd6b29eb0f2"} Oct 30 02:30:03.331928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.331911 5035 scope.go:122] "RemoveContainer" containerID="cd06c2e14767e6dd6cd067a9577ecd93552407a3d05f85b955612bd6b29eb0f2" Oct 30 02:30:03.333951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.333914 5035 generic.go:356] "Generic (PLEG): container finished" podID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerID="ae16ffa44ac7f724adaea234be0b08ce896a055fb33d992af47bf8660ddf00ec" exitCode=60 Oct 30 02:30:03.334064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.334044 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" event={"ID":"e8bb030d-d1b0-4485-8d30-3fbdfa293226","Type":"ContainerDied","Data":"ae16ffa44ac7f724adaea234be0b08ce896a055fb33d992af47bf8660ddf00ec"} Oct 30 02:30:03.334305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.334287 5035 scope.go:122] "RemoveContainer" containerID="ae16ffa44ac7f724adaea234be0b08ce896a055fb33d992af47bf8660ddf00ec" Oct 30 02:30:03.336074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.336049 5035 generic.go:356] "Generic (PLEG): container finished" podID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerID="490ec57b0fbeb44c97907ce40aa8a7fd8b949da7e79d2ef1c835dda41d06f3d1" exitCode=60 Oct 30 02:30:03.336134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.336118 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" event={"ID":"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4","Type":"ContainerDied","Data":"490ec57b0fbeb44c97907ce40aa8a7fd8b949da7e79d2ef1c835dda41d06f3d1"} Oct 30 02:30:03.336361 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.336344 5035 scope.go:122] "RemoveContainer" containerID="490ec57b0fbeb44c97907ce40aa8a7fd8b949da7e79d2ef1c835dda41d06f3d1" Oct 30 02:30:03.337972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.337941 5035 generic.go:356] "Generic (PLEG): container finished" podID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerID="90532f499a5deb230be510fc053e926ab5388a4510b6ebcfbcfa0e63052226fd" exitCode=60 Oct 30 02:30:03.338066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.338004 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" event={"ID":"103ec715-4f09-41c8-adba-c72457e4e9e1","Type":"ContainerDied","Data":"90532f499a5deb230be510fc053e926ab5388a4510b6ebcfbcfa0e63052226fd"} Oct 30 02:30:03.338265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.338242 5035 scope.go:122] "RemoveContainer" containerID="90532f499a5deb230be510fc053e926ab5388a4510b6ebcfbcfa0e63052226fd" Oct 30 02:30:03.339836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.339817 5035 generic.go:356] "Generic (PLEG): container finished" podID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerID="ac6a76fea4c1fed130a73f7ae51cde9c80c8229eefb7ab64e0b488e93c6c8335" exitCode=60 Oct 30 02:30:03.339951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.339862 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" event={"ID":"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12","Type":"ContainerDied","Data":"ac6a76fea4c1fed130a73f7ae51cde9c80c8229eefb7ab64e0b488e93c6c8335"} Oct 30 02:30:03.340156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:03.340140 5035 scope.go:122] "RemoveContainer" containerID="ac6a76fea4c1fed130a73f7ae51cde9c80c8229eefb7ab64e0b488e93c6c8335" Oct 30 02:30:04.353374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.353333 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" event={"ID":"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12","Type":"ContainerStarted","Data":"ecccb3e0b881bf380f0dd344d37bc153eb96b9b0299c95d1f46f92cde2ec7246"} Oct 30 02:30:04.355796 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.355757 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" event={"ID":"bf2a4698-c3d7-4af1-9903-76566281a8ec","Type":"ContainerStarted","Data":"1ddb7060de9e23e553c322bc5992dcedd15b5adf899012f82607f0fd41ef995f"} Oct 30 02:30:04.358083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.358053 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" event={"ID":"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da","Type":"ContainerStarted","Data":"03dd29afd9b2cc89d4af3caf6046424f88022fe7260984ed9e081b7de1ca83be"} Oct 30 02:30:04.360131 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.360100 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" event={"ID":"e8bb030d-d1b0-4485-8d30-3fbdfa293226","Type":"ContainerStarted","Data":"7a7a8d7979d2b47bb6764dba55662f0ae083c71e2f6657759aa00a139404143e"} Oct 30 02:30:04.362249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.362221 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" event={"ID":"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4","Type":"ContainerStarted","Data":"97423523d66a673d75e0c533dd9a95212b8cebf457b0cacd82e90168ef5924fa"} Oct 30 02:30:04.364298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.364270 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" event={"ID":"103ec715-4f09-41c8-adba-c72457e4e9e1","Type":"ContainerStarted","Data":"e1fc2ed74459108f0b1ddf557e5ff81274b4472dfa43c2322c75dca62e08ef9e"} Oct 30 02:30:04.402352 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.402287 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" podStartSLOduration=4.4022712 podStartE2EDuration="4.4022712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.396604142 +0000 UTC m=+3682.899536361" watchObservedRunningTime="2026-10-30 02:30:04.4022712 +0000 UTC m=+3682.905203419" Oct 30 02:30:04.444934 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.444871 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" podStartSLOduration=4.444853903 podStartE2EDuration="4.444853903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.442832905 +0000 UTC m=+3682.945765125" watchObservedRunningTime="2026-10-30 02:30:04.444853903 +0000 UTC m=+3682.947786123" Oct 30 02:30:04.492730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.492673 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" podStartSLOduration=4.492658666 podStartE2EDuration="4.492658666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.486906196 +0000 UTC m=+3682.989838418" watchObservedRunningTime="2026-10-30 02:30:04.492658666 +0000 UTC m=+3682.995590886" Oct 30 02:30:04.533648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.533506 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" podStartSLOduration=4.533490421 podStartE2EDuration="4.533490421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.530382688 +0000 UTC m=+3683.033314909" watchObservedRunningTime="2026-10-30 02:30:04.533490421 +0000 UTC m=+3683.036422641" Oct 30 02:30:04.577600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.577505 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" podStartSLOduration=4.577398413 podStartE2EDuration="4.577398413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.576187053 +0000 UTC m=+3683.079119273" watchObservedRunningTime="2026-10-30 02:30:04.577398413 +0000 UTC m=+3683.080330640" Oct 30 02:30:04.624918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:04.624850 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" podStartSLOduration=4.624832018 podStartE2EDuration="4.624832018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.61956523 +0000 UTC m=+3683.122497452" watchObservedRunningTime="2026-10-30 02:30:04.624832018 +0000 UTC m=+3683.127764241" Oct 30 02:30:05.375445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.375401 5035 generic.go:356] "Generic (PLEG): container finished" podID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerID="e1fc2ed74459108f0b1ddf557e5ff81274b4472dfa43c2322c75dca62e08ef9e" exitCode=60 Oct 30 02:30:05.375958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.375479 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" event={"ID":"103ec715-4f09-41c8-adba-c72457e4e9e1","Type":"ContainerDied","Data":"e1fc2ed74459108f0b1ddf557e5ff81274b4472dfa43c2322c75dca62e08ef9e"} Oct 30 02:30:05.375958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.375591 5035 scope.go:122] "RemoveContainer" containerID="90532f499a5deb230be510fc053e926ab5388a4510b6ebcfbcfa0e63052226fd" Oct 30 02:30:05.376088 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.375992 5035 scope.go:122] "RemoveContainer" containerID="e1fc2ed74459108f0b1ddf557e5ff81274b4472dfa43c2322c75dca62e08ef9e" Oct 30 02:30:05.376427 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:05.376402 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s_odh-ai-gateway-infra(103ec715-4f09-41c8-adba-c72457e4e9e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" Oct 30 02:30:05.377871 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.377849 5035 generic.go:356] "Generic (PLEG): container finished" podID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerID="ecccb3e0b881bf380f0dd344d37bc153eb96b9b0299c95d1f46f92cde2ec7246" exitCode=60 Oct 30 02:30:05.377973 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.377934 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" event={"ID":"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12","Type":"ContainerDied","Data":"ecccb3e0b881bf380f0dd344d37bc153eb96b9b0299c95d1f46f92cde2ec7246"} Oct 30 02:30:05.378270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.378250 5035 scope.go:122] "RemoveContainer" containerID="ecccb3e0b881bf380f0dd344d37bc153eb96b9b0299c95d1f46f92cde2ec7246" Oct 30 02:30:05.378684 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:05.378663 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x_odh-ai-gateway-infra(540dd1e7-ebaf-4d58-83a1-fc2334bbfa12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" Oct 30 02:30:05.379966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.379944 5035 generic.go:356] "Generic (PLEG): container finished" podID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerID="1ddb7060de9e23e553c322bc5992dcedd15b5adf899012f82607f0fd41ef995f" exitCode=60 Oct 30 02:30:05.380152 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.380022 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" event={"ID":"bf2a4698-c3d7-4af1-9903-76566281a8ec","Type":"ContainerDied","Data":"1ddb7060de9e23e553c322bc5992dcedd15b5adf899012f82607f0fd41ef995f"} Oct 30 02:30:05.380447 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.380423 5035 scope.go:122] "RemoveContainer" containerID="1ddb7060de9e23e553c322bc5992dcedd15b5adf899012f82607f0fd41ef995f" Oct 30 02:30:05.380904 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:05.380881 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7_odh-ai-gateway-infra(bf2a4698-c3d7-4af1-9903-76566281a8ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" Oct 30 02:30:05.382733 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.382712 5035 generic.go:356] "Generic (PLEG): container finished" podID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerID="03dd29afd9b2cc89d4af3caf6046424f88022fe7260984ed9e081b7de1ca83be" exitCode=60 Oct 30 02:30:05.382827 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.382747 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" event={"ID":"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da","Type":"ContainerDied","Data":"03dd29afd9b2cc89d4af3caf6046424f88022fe7260984ed9e081b7de1ca83be"} Oct 30 02:30:05.383172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.383153 5035 scope.go:122] "RemoveContainer" containerID="03dd29afd9b2cc89d4af3caf6046424f88022fe7260984ed9e081b7de1ca83be" Oct 30 02:30:05.383635 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:05.383608 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt_odh-ai-gateway-infra(b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" Oct 30 02:30:05.384857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.384834 5035 generic.go:356] "Generic (PLEG): container finished" podID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerID="7a7a8d7979d2b47bb6764dba55662f0ae083c71e2f6657759aa00a139404143e" exitCode=60 Oct 30 02:30:05.385048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.385030 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" event={"ID":"e8bb030d-d1b0-4485-8d30-3fbdfa293226","Type":"ContainerDied","Data":"7a7a8d7979d2b47bb6764dba55662f0ae083c71e2f6657759aa00a139404143e"} Oct 30 02:30:05.385433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.385413 5035 scope.go:122] "RemoveContainer" containerID="7a7a8d7979d2b47bb6764dba55662f0ae083c71e2f6657759aa00a139404143e" Oct 30 02:30:05.385800 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:05.385777 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm_odh-ai-gateway-infra(e8bb030d-d1b0-4485-8d30-3fbdfa293226)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" Oct 30 02:30:05.388111 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.388090 5035 generic.go:356] "Generic (PLEG): container finished" podID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerID="97423523d66a673d75e0c533dd9a95212b8cebf457b0cacd82e90168ef5924fa" exitCode=60 Oct 30 02:30:05.388225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.388156 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" event={"ID":"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4","Type":"ContainerDied","Data":"97423523d66a673d75e0c533dd9a95212b8cebf457b0cacd82e90168ef5924fa"} Oct 30 02:30:05.388419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.388403 5035 scope.go:122] "RemoveContainer" containerID="97423523d66a673d75e0c533dd9a95212b8cebf457b0cacd82e90168ef5924fa" Oct 30 02:30:05.388751 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:05.388733 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r_odh-ai-gateway-infra(0834bdb9-90d6-447f-bc74-dba6cc5cdfa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" Oct 30 02:30:05.392353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.392335 5035 scope.go:122] "RemoveContainer" containerID="ac6a76fea4c1fed130a73f7ae51cde9c80c8229eefb7ab64e0b488e93c6c8335" Oct 30 02:30:05.408691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.408664 5035 scope.go:122] "RemoveContainer" containerID="91031bcb2a3dc531b622488cc1de01fd40fbdfce71073cb7ab007c442af5f751" Oct 30 02:30:05.426694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.426661 5035 scope.go:122] "RemoveContainer" containerID="cd06c2e14767e6dd6cd067a9577ecd93552407a3d05f85b955612bd6b29eb0f2" Oct 30 02:30:05.447584 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.447521 5035 scope.go:122] "RemoveContainer" containerID="ae16ffa44ac7f724adaea234be0b08ce896a055fb33d992af47bf8660ddf00ec" Oct 30 02:30:05.467595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:05.467559 5035 scope.go:122] "RemoveContainer" containerID="490ec57b0fbeb44c97907ce40aa8a7fd8b949da7e79d2ef1c835dda41d06f3d1" Oct 30 02:30:06.400846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:06.400813 5035 scope.go:122] "RemoveContainer" containerID="97423523d66a673d75e0c533dd9a95212b8cebf457b0cacd82e90168ef5924fa" Oct 30 02:30:06.401347 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:06.401284 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r_odh-ai-gateway-infra(0834bdb9-90d6-447f-bc74-dba6cc5cdfa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" Oct 30 02:30:06.402690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:06.402660 5035 scope.go:122] "RemoveContainer" containerID="e1fc2ed74459108f0b1ddf557e5ff81274b4472dfa43c2322c75dca62e08ef9e" Oct 30 02:30:06.403132 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:06.403104 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s_odh-ai-gateway-infra(103ec715-4f09-41c8-adba-c72457e4e9e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" Oct 30 02:30:06.404779 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:06.404749 5035 scope.go:122] "RemoveContainer" containerID="ecccb3e0b881bf380f0dd344d37bc153eb96b9b0299c95d1f46f92cde2ec7246" Oct 30 02:30:06.405137 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:06.405115 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x_odh-ai-gateway-infra(540dd1e7-ebaf-4d58-83a1-fc2334bbfa12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" Oct 30 02:30:06.406621 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:06.406593 5035 scope.go:122] "RemoveContainer" containerID="1ddb7060de9e23e553c322bc5992dcedd15b5adf899012f82607f0fd41ef995f" Oct 30 02:30:06.406950 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:06.406922 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7_odh-ai-gateway-infra(bf2a4698-c3d7-4af1-9903-76566281a8ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" Oct 30 02:30:06.408273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:06.408250 5035 scope.go:122] "RemoveContainer" containerID="03dd29afd9b2cc89d4af3caf6046424f88022fe7260984ed9e081b7de1ca83be" Oct 30 02:30:06.408603 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:06.408583 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt_odh-ai-gateway-infra(b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" Oct 30 02:30:06.409947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:06.409929 5035 scope.go:122] "RemoveContainer" containerID="7a7a8d7979d2b47bb6764dba55662f0ae083c71e2f6657759aa00a139404143e" Oct 30 02:30:06.410235 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:06.410215 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm_odh-ai-gateway-infra(e8bb030d-d1b0-4485-8d30-3fbdfa293226)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" Oct 30 02:30:15.670555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:15.655786 5035 scope.go:122] "RemoveContainer" containerID="1ddb7060de9e23e553c322bc5992dcedd15b5adf899012f82607f0fd41ef995f" Oct 30 02:30:16.512974 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:16.512933 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" event={"ID":"bf2a4698-c3d7-4af1-9903-76566281a8ec","Type":"ContainerStarted","Data":"437c836e2c86dcf34ac6c778cdafc2ff9c47fb67fb3a273c8f6351ca77aa6d40"} Oct 30 02:30:16.636182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:16.636144 5035 scope.go:122] "RemoveContainer" containerID="e1fc2ed74459108f0b1ddf557e5ff81274b4472dfa43c2322c75dca62e08ef9e" Oct 30 02:30:17.524520 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.524429 5035 generic.go:356] "Generic (PLEG): container finished" podID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerID="437c836e2c86dcf34ac6c778cdafc2ff9c47fb67fb3a273c8f6351ca77aa6d40" exitCode=60 Oct 30 02:30:17.524520 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.524505 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" event={"ID":"bf2a4698-c3d7-4af1-9903-76566281a8ec","Type":"ContainerDied","Data":"437c836e2c86dcf34ac6c778cdafc2ff9c47fb67fb3a273c8f6351ca77aa6d40"} Oct 30 02:30:17.525059 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.524584 5035 scope.go:122] "RemoveContainer" containerID="1ddb7060de9e23e553c322bc5992dcedd15b5adf899012f82607f0fd41ef995f" Oct 30 02:30:17.525100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.525067 5035 scope.go:122] "RemoveContainer" containerID="437c836e2c86dcf34ac6c778cdafc2ff9c47fb67fb3a273c8f6351ca77aa6d40" Oct 30 02:30:17.525521 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:17.525501 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7_odh-ai-gateway-infra(bf2a4698-c3d7-4af1-9903-76566281a8ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" Oct 30 02:30:17.533294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.533260 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" event={"ID":"103ec715-4f09-41c8-adba-c72457e4e9e1","Type":"ContainerStarted","Data":"45d8a4a50edd2c811d8015dd7ca41a75d873103ec444239ac9e64ac7d4846b78"} Oct 30 02:30:17.574457 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.574415 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7"] Oct 30 02:30:17.636705 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:17.635576 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:30:17.636705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.636125 5035 scope.go:122] "RemoveContainer" containerID="97423523d66a673d75e0c533dd9a95212b8cebf457b0cacd82e90168ef5924fa" Oct 30 02:30:17.636705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.636384 5035 scope.go:122] "RemoveContainer" containerID="ecccb3e0b881bf380f0dd344d37bc153eb96b9b0299c95d1f46f92cde2ec7246" Oct 30 02:30:17.636705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.636466 5035 scope.go:122] "RemoveContainer" containerID="03dd29afd9b2cc89d4af3caf6046424f88022fe7260984ed9e081b7de1ca83be" Oct 30 02:30:17.636705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:17.636605 5035 scope.go:122] "RemoveContainer" containerID="7a7a8d7979d2b47bb6764dba55662f0ae083c71e2f6657759aa00a139404143e" Oct 30 02:30:18.294370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.294331 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf"] Oct 30 02:30:18.300522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.300490 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.319959 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.319926 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4593af-openshift-default-dockercfg-cnzsh\"" Oct 30 02:30:18.323936 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.323888 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf"] Oct 30 02:30:18.424181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.424136 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-data\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.424398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.424259 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-istio-token\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.424398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.424313 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-envoy\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.424492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.424442 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0175afca-7f43-471c-9ffb-bd96f1150806-istiod-ca-cert\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.424579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.424558 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-certs\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.424656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.424620 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.424734 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.424720 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-credential-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.424854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.424838 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpv8\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-kube-api-access-jhpv8\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.424905 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.424893 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0175afca-7f43-471c-9ffb-bd96f1150806-istio-podinfo\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.526545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.526489 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-data\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.526635 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-istio-token\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.526678 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-envoy\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.526728 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0175afca-7f43-471c-9ffb-bd96f1150806-istiod-ca-cert\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.526884 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-certs\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.526956 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.526897 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-data\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.527019 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-credential-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.527109 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-certs\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.527206 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpv8\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-kube-api-access-jhpv8\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.527310 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0175afca-7f43-471c-9ffb-bd96f1150806-istio-podinfo\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.527321 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.527434 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-credential-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.527789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.527506 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0175afca-7f43-471c-9ffb-bd96f1150806-istiod-ca-cert\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.529232 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.529209 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-envoy\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.529727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.529710 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0175afca-7f43-471c-9ffb-bd96f1150806-istio-podinfo\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.545846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.545806 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" event={"ID":"e8bb030d-d1b0-4485-8d30-3fbdfa293226","Type":"ContainerStarted","Data":"b0b60834ebef76e224f94734c90e05fabe1649f95081d5d9a87b10c8027b0969"} Oct 30 02:30:18.548428 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.548392 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" event={"ID":"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4","Type":"ContainerStarted","Data":"eb37574c54ce6f379832b800a4a3320bcf49d3e478ccb9d50d9ad49f9a67d4b6"} Oct 30 02:30:18.551792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.551754 5035 generic.go:356] "Generic (PLEG): container finished" podID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerID="45d8a4a50edd2c811d8015dd7ca41a75d873103ec444239ac9e64ac7d4846b78" exitCode=60 Oct 30 02:30:18.552063 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.552039 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" event={"ID":"103ec715-4f09-41c8-adba-c72457e4e9e1","Type":"ContainerDied","Data":"45d8a4a50edd2c811d8015dd7ca41a75d873103ec444239ac9e64ac7d4846b78"} Oct 30 02:30:18.552124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.552103 5035 scope.go:122] "RemoveContainer" containerID="e1fc2ed74459108f0b1ddf557e5ff81274b4472dfa43c2322c75dca62e08ef9e" Oct 30 02:30:18.553156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.552587 5035 scope.go:122] "RemoveContainer" containerID="45d8a4a50edd2c811d8015dd7ca41a75d873103ec444239ac9e64ac7d4846b78" Oct 30 02:30:18.553156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.552849 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpv8\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-kube-api-access-jhpv8\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.553156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.552871 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-istio-token\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.553156 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:18.553053 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s_odh-ai-gateway-infra(103ec715-4f09-41c8-adba-c72457e4e9e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" Oct 30 02:30:18.555157 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.555134 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" event={"ID":"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12","Type":"ContainerStarted","Data":"33f39ea8170cbfee536f574094347087c3fde71eca749244959e0cc436b7e9eb"} Oct 30 02:30:18.560631 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.560596 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" event={"ID":"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da","Type":"ContainerStarted","Data":"64bef96866d0e7999ad3335e31ec2928024c0e594ce2b11fa560cf084e5cbd08"} Oct 30 02:30:18.618782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.618742 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:18.665868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.665829 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s"] Oct 30 02:30:18.964625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:18.964576 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:19.100887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.097497 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf2a4698-c3d7-4af1-9903-76566281a8ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2a4698-c3d7-4af1-9903-76566281a8ec-maas-api-service-ca\") pod \"bf2a4698-c3d7-4af1-9903-76566281a8ec\" (UID: \"bf2a4698-c3d7-4af1-9903-76566281a8ec\") " Oct 30 02:30:19.100887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.097778 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2a4698-c3d7-4af1-9903-76566281a8ec-kube-api-access-jlz9z\" (UniqueName: \"kubernetes.io/projected/bf2a4698-c3d7-4af1-9903-76566281a8ec-kube-api-access-jlz9z\") pod \"bf2a4698-c3d7-4af1-9903-76566281a8ec\" (UID: \"bf2a4698-c3d7-4af1-9903-76566281a8ec\") " Oct 30 02:30:19.109047 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.108260 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2a4698-c3d7-4af1-9903-76566281a8ec-maas-api-service-ca" pod "bf2a4698-c3d7-4af1-9903-76566281a8ec" (UID: "bf2a4698-c3d7-4af1-9903-76566281a8ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:19.148248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.148192 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2a4698-c3d7-4af1-9903-76566281a8ec-kube-api-access-jlz9z" pod "bf2a4698-c3d7-4af1-9903-76566281a8ec" (UID: "bf2a4698-c3d7-4af1-9903-76566281a8ec"). InnerVolumeSpecName "kube-api-access-jlz9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:19.200660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.200613 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2a4698-c3d7-4af1-9903-76566281a8ec-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:19.200660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.200649 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlz9z\" (UniqueName: \"kubernetes.io/projected/bf2a4698-c3d7-4af1-9903-76566281a8ec-kube-api-access-jlz9z\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:19.342211 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.342174 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf"] Oct 30 02:30:19.342359 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:30:19.342318 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0175afca_7f43_471c_9ffb_bd96f1150806.slice/crio-6d32a31780e93623a401a9177a0cc6eb2f4ea0d060d9db7dd4d7d823d1b57f1b WatchSource:0}: Error finding container 6d32a31780e93623a401a9177a0cc6eb2f4ea0d060d9db7dd4d7d823d1b57f1b: Status 404 returned error can't find the container with id 6d32a31780e93623a401a9177a0cc6eb2f4ea0d060d9db7dd4d7d823d1b57f1b Oct 30 02:30:19.346334 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.346291 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:30:19.346442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.346396 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:30:19.346483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.346441 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:30:19.576075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.576035 5035 generic.go:356] "Generic (PLEG): container finished" podID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerID="b0b60834ebef76e224f94734c90e05fabe1649f95081d5d9a87b10c8027b0969" exitCode=60 Oct 30 02:30:19.576640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.576104 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" event={"ID":"e8bb030d-d1b0-4485-8d30-3fbdfa293226","Type":"ContainerDied","Data":"b0b60834ebef76e224f94734c90e05fabe1649f95081d5d9a87b10c8027b0969"} Oct 30 02:30:19.576640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.576173 5035 scope.go:122] "RemoveContainer" containerID="7a7a8d7979d2b47bb6764dba55662f0ae083c71e2f6657759aa00a139404143e" Oct 30 02:30:19.576778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.576719 5035 scope.go:122] "RemoveContainer" containerID="b0b60834ebef76e224f94734c90e05fabe1649f95081d5d9a87b10c8027b0969" Oct 30 02:30:19.577246 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:19.577208 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm_odh-ai-gateway-infra(e8bb030d-d1b0-4485-8d30-3fbdfa293226)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" Oct 30 02:30:19.578969 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.578945 5035 generic.go:356] "Generic (PLEG): container finished" podID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerID="eb37574c54ce6f379832b800a4a3320bcf49d3e478ccb9d50d9ad49f9a67d4b6" exitCode=60 Oct 30 02:30:19.579098 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.578994 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" event={"ID":"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4","Type":"ContainerDied","Data":"eb37574c54ce6f379832b800a4a3320bcf49d3e478ccb9d50d9ad49f9a67d4b6"} Oct 30 02:30:19.579548 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.579511 5035 scope.go:122] "RemoveContainer" containerID="eb37574c54ce6f379832b800a4a3320bcf49d3e478ccb9d50d9ad49f9a67d4b6" Oct 30 02:30:19.582694 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:19.579999 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r_odh-ai-gateway-infra(0834bdb9-90d6-447f-bc74-dba6cc5cdfa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" Oct 30 02:30:19.586042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.585295 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" event={"ID":"0175afca-7f43-471c-9ffb-bd96f1150806","Type":"ContainerStarted","Data":"dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80"} Oct 30 02:30:19.586042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.585354 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" event={"ID":"0175afca-7f43-471c-9ffb-bd96f1150806","Type":"ContainerStarted","Data":"6d32a31780e93623a401a9177a0cc6eb2f4ea0d060d9db7dd4d7d823d1b57f1b"} Oct 30 02:30:19.589602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.589507 5035 generic.go:356] "Generic (PLEG): container finished" podID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerID="33f39ea8170cbfee536f574094347087c3fde71eca749244959e0cc436b7e9eb" exitCode=60 Oct 30 02:30:19.589602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.589587 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" event={"ID":"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12","Type":"ContainerDied","Data":"33f39ea8170cbfee536f574094347087c3fde71eca749244959e0cc436b7e9eb"} Oct 30 02:30:19.591024 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.590072 5035 scope.go:122] "RemoveContainer" containerID="33f39ea8170cbfee536f574094347087c3fde71eca749244959e0cc436b7e9eb" Oct 30 02:30:19.591024 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:19.590471 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x_odh-ai-gateway-infra(540dd1e7-ebaf-4d58-83a1-fc2334bbfa12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" Oct 30 02:30:19.591632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.591503 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" Oct 30 02:30:19.591632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.591517 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7" event={"ID":"bf2a4698-c3d7-4af1-9903-76566281a8ec","Type":"ContainerDied","Data":"c75ca8ffdd8362a67b7e0ff3f427ceacc3c727378d26db5b7a7cf7de06bedab2"} Oct 30 02:30:19.597704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.597669 5035 generic.go:356] "Generic (PLEG): container finished" podID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerID="64bef96866d0e7999ad3335e31ec2928024c0e594ce2b11fa560cf084e5cbd08" exitCode=60 Oct 30 02:30:19.597857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.597726 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" event={"ID":"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da","Type":"ContainerDied","Data":"64bef96866d0e7999ad3335e31ec2928024c0e594ce2b11fa560cf084e5cbd08"} Oct 30 02:30:19.598277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.598253 5035 scope.go:122] "RemoveContainer" containerID="64bef96866d0e7999ad3335e31ec2928024c0e594ce2b11fa560cf084e5cbd08" Oct 30 02:30:19.598852 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:19.598823 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt_odh-ai-gateway-infra(b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" Oct 30 02:30:19.603339 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.603313 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm"] Oct 30 02:30:19.607662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.607637 5035 scope.go:122] "RemoveContainer" containerID="97423523d66a673d75e0c533dd9a95212b8cebf457b0cacd82e90168ef5924fa" Oct 30 02:30:19.619136 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.618860 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:19.633798 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.633765 5035 scope.go:122] "RemoveContainer" containerID="ecccb3e0b881bf380f0dd344d37bc153eb96b9b0299c95d1f46f92cde2ec7246" Oct 30 02:30:19.653863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.653831 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt"] Oct 30 02:30:19.665911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.665853 5035 scope.go:122] "RemoveContainer" containerID="437c836e2c86dcf34ac6c778cdafc2ff9c47fb67fb3a273c8f6351ca77aa6d40" Oct 30 02:30:19.689580 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.680718 5035 scope.go:122] "RemoveContainer" containerID="03dd29afd9b2cc89d4af3caf6046424f88022fe7260984ed9e081b7de1ca83be" Oct 30 02:30:19.714801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.713876 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r"] Oct 30 02:30:19.761892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.761822 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" podStartSLOduration=1.761796114 podStartE2EDuration="1.761796114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:19.739000605 +0000 UTC m=+3698.241932825" watchObservedRunningTime="2026-10-30 02:30:19.761796114 +0000 UTC m=+3698.264728335" Oct 30 02:30:19.776743 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.776709 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x"] Oct 30 02:30:19.794995 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.794963 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:19.926100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.926014 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103ec715-4f09-41c8-adba-c72457e4e9e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103ec715-4f09-41c8-adba-c72457e4e9e1-maas-api-service-ca\") pod \"103ec715-4f09-41c8-adba-c72457e4e9e1\" (UID: \"103ec715-4f09-41c8-adba-c72457e4e9e1\") " Oct 30 02:30:19.926284 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.926269 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103ec715-4f09-41c8-adba-c72457e4e9e1-kube-api-access-7lzw6\" (UniqueName: \"kubernetes.io/projected/103ec715-4f09-41c8-adba-c72457e4e9e1-kube-api-access-7lzw6\") pod \"103ec715-4f09-41c8-adba-c72457e4e9e1\" (UID: \"103ec715-4f09-41c8-adba-c72457e4e9e1\") " Oct 30 02:30:19.926385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.926357 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103ec715-4f09-41c8-adba-c72457e4e9e1-maas-api-service-ca" pod "103ec715-4f09-41c8-adba-c72457e4e9e1" (UID: "103ec715-4f09-41c8-adba-c72457e4e9e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:19.926789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.926764 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/103ec715-4f09-41c8-adba-c72457e4e9e1-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:19.927811 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.927785 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7"] Oct 30 02:30:19.928570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.928519 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103ec715-4f09-41c8-adba-c72457e4e9e1-kube-api-access-7lzw6" pod "103ec715-4f09-41c8-adba-c72457e4e9e1" (UID: "103ec715-4f09-41c8-adba-c72457e4e9e1"). InnerVolumeSpecName "kube-api-access-7lzw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:19.934001 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:19.933966 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888790-75tr7"] Oct 30 02:30:20.028511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:20.028476 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lzw6\" (UniqueName: \"kubernetes.io/projected/103ec715-4f09-41c8-adba-c72457e4e9e1-kube-api-access-7lzw6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:20.632559 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:20.629707 5035 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 02:30:20.632559 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:20.629780 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 02:30:20.765845 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:20.765810 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" Oct 30 02:30:20.766032 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:20.765947 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s" event={"ID":"103ec715-4f09-41c8-adba-c72457e4e9e1","Type":"ContainerDied","Data":"3ce3cc3817464a2b4960aff1d68382420ba77867c4783aa8f1d5183f0b95a9c6"} Oct 30 02:30:20.766032 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:20.766006 5035 scope.go:122] "RemoveContainer" containerID="45d8a4a50edd2c811d8015dd7ca41a75d873103ec444239ac9e64ac7d4846b78" Oct 30 02:30:21.017875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.014011 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s"] Oct 30 02:30:21.022485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.022445 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888790-jln5s"] Oct 30 02:30:21.255598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.254938 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:21.422596 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.422565 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:21.422838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.422810 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-kube-api-access-jtwfj\" (UniqueName: \"kubernetes.io/projected/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-kube-api-access-jtwfj\") pod \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\" (UID: \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\") " Oct 30 02:30:21.422988 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.422971 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-maas-api-service-ca\") pod \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\" (UID: \"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12\") " Oct 30 02:30:21.424124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.424094 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-maas-api-service-ca" pod "540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" (UID: "540dd1e7-ebaf-4d58-83a1-fc2334bbfa12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:21.425886 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.425855 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-kube-api-access-jtwfj" pod "540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" (UID: "540dd1e7-ebaf-4d58-83a1-fc2334bbfa12"). InnerVolumeSpecName "kube-api-access-jtwfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:21.433409 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.433373 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:21.451206 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.451170 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:21.537238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.532448 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-maas-api-service-ca\") pod \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\" (UID: \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\") " Oct 30 02:30:21.537238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.532909 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-maas-api-service-ca" pod "0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" (UID: "0834bdb9-90d6-447f-bc74-dba6cc5cdfa4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:21.537238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.533057 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-kube-api-access-tdgvj\" (UniqueName: \"kubernetes.io/projected/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-kube-api-access-tdgvj\") pod \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\" (UID: \"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4\") " Oct 30 02:30:21.537238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.533633 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bb030d-d1b0-4485-8d30-3fbdfa293226-kube-api-access-n4krt\" (UniqueName: \"kubernetes.io/projected/e8bb030d-d1b0-4485-8d30-3fbdfa293226-kube-api-access-n4krt\") pod \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\" (UID: \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\") " Oct 30 02:30:21.537238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.534165 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-maas-api-service-ca\") pod \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\" (UID: \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\") " Oct 30 02:30:21.537238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.534627 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-maas-api-service-ca" pod "b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" (UID: "b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:21.537238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.534736 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bb030d-d1b0-4485-8d30-3fbdfa293226-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bb030d-d1b0-4485-8d30-3fbdfa293226-maas-api-service-ca\") pod \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\" (UID: \"e8bb030d-d1b0-4485-8d30-3fbdfa293226\") " Oct 30 02:30:21.537238 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.534982 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-kube-api-access-m2b2q\" (UniqueName: \"kubernetes.io/projected/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-kube-api-access-m2b2q\") pod \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\" (UID: \"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da\") " Oct 30 02:30:21.538241 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.538212 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bb030d-d1b0-4485-8d30-3fbdfa293226-maas-api-service-ca" pod "e8bb030d-d1b0-4485-8d30-3fbdfa293226" (UID: "e8bb030d-d1b0-4485-8d30-3fbdfa293226"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:21.539624 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.539596 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:21.540369 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.540343 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:21.540713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.540614 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtwfj\" (UniqueName: \"kubernetes.io/projected/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12-kube-api-access-jtwfj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:21.540713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.540636 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:21.540713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.540654 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bb030d-d1b0-4485-8d30-3fbdfa293226-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:21.541731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.541648 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bb030d-d1b0-4485-8d30-3fbdfa293226-kube-api-access-n4krt" pod "e8bb030d-d1b0-4485-8d30-3fbdfa293226" (UID: "e8bb030d-d1b0-4485-8d30-3fbdfa293226"). InnerVolumeSpecName "kube-api-access-n4krt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:21.542230 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.542178 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-kube-api-access-m2b2q" pod "b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" (UID: "b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da"). InnerVolumeSpecName "kube-api-access-m2b2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:21.542803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.542772 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-kube-api-access-tdgvj" pod "0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" (UID: "0834bdb9-90d6-447f-bc74-dba6cc5cdfa4"). InnerVolumeSpecName "kube-api-access-tdgvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:21.620006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.619906 5035 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 02:30:21.620006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.619973 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 02:30:21.643230 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.643151 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4krt\" (UniqueName: \"kubernetes.io/projected/e8bb030d-d1b0-4485-8d30-3fbdfa293226-kube-api-access-n4krt\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:21.643230 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.643190 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2b2q\" (UniqueName: \"kubernetes.io/projected/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da-kube-api-access-m2b2q\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:21.643230 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.643205 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdgvj\" (UniqueName: \"kubernetes.io/projected/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4-kube-api-access-tdgvj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:21.651655 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.651613 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" path="/var/lib/kubelet/pods/103ec715-4f09-41c8-adba-c72457e4e9e1/volumes" Oct 30 02:30:21.652294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.652274 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" path="/var/lib/kubelet/pods/bf2a4698-c3d7-4af1-9903-76566281a8ec/volumes" Oct 30 02:30:21.804909 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.804878 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" Oct 30 02:30:21.805100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.804881 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt" event={"ID":"b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da","Type":"ContainerDied","Data":"8db2b91b69af58e4afb8ba2b11150785ca21ad8436276bf7d97f6f3e2c65a282"} Oct 30 02:30:21.805100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.805036 5035 scope.go:122] "RemoveContainer" containerID="64bef96866d0e7999ad3335e31ec2928024c0e594ce2b11fa560cf084e5cbd08" Oct 30 02:30:21.806685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.806660 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" Oct 30 02:30:21.806685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.806671 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm" event={"ID":"e8bb030d-d1b0-4485-8d30-3fbdfa293226","Type":"ContainerDied","Data":"1f82b8707ec5cff3edd2fae0eada569be27ed03feb94957dd6555f0949c03f27"} Oct 30 02:30:21.808656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.808634 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" Oct 30 02:30:21.808775 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.808675 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r" event={"ID":"0834bdb9-90d6-447f-bc74-dba6cc5cdfa4","Type":"ContainerDied","Data":"e4b88a26ced8b94c9f2552c61f85611b91712ef5192d1ad16f9338e52f673574"} Oct 30 02:30:21.812257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.812230 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" Oct 30 02:30:21.812257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.812248 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x" event={"ID":"540dd1e7-ebaf-4d58-83a1-fc2334bbfa12","Type":"ContainerDied","Data":"e2f9bf194b6e1341371cdd24859d4c291c66ba9c361b766b59935a1f30417271"} Oct 30 02:30:21.818871 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.818849 5035 scope.go:122] "RemoveContainer" containerID="b0b60834ebef76e224f94734c90e05fabe1649f95081d5d9a87b10c8027b0969" Oct 30 02:30:21.830393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.830362 5035 scope.go:122] "RemoveContainer" containerID="eb37574c54ce6f379832b800a4a3320bcf49d3e478ccb9d50d9ad49f9a67d4b6" Oct 30 02:30:21.841859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.841832 5035 scope.go:122] "RemoveContainer" containerID="33f39ea8170cbfee536f574094347087c3fde71eca749244959e0cc436b7e9eb" Oct 30 02:30:21.893621 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.892630 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt"] Oct 30 02:30:21.899996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.899954 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888790-9c2gt"] Oct 30 02:30:21.925801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.925762 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r"] Oct 30 02:30:21.931745 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:21.931708 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888790-jx87r"] Oct 30 02:30:22.002481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:22.002445 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x"] Oct 30 02:30:22.007311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:22.007279 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888790-bph6x"] Oct 30 02:30:22.058256 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:22.058218 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm"] Oct 30 02:30:22.063411 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:22.063379 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888790-kx6lm"] Oct 30 02:30:22.620071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:22.620020 5035 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 02:30:22.620340 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:22.620102 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 02:30:23.619841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:23.619801 5035 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 02:30:23.620292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:23.619863 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 02:30:23.646758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:23.646724 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" path="/var/lib/kubelet/pods/0834bdb9-90d6-447f-bc74-dba6cc5cdfa4/volumes" Oct 30 02:30:23.647314 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:23.647299 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" path="/var/lib/kubelet/pods/540dd1e7-ebaf-4d58-83a1-fc2334bbfa12/volumes" Oct 30 02:30:23.647826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:23.647811 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" path="/var/lib/kubelet/pods/b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da/volumes" Oct 30 02:30:23.648639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:23.648624 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" path="/var/lib/kubelet/pods/e8bb030d-d1b0-4485-8d30-3fbdfa293226/volumes" Oct 30 02:30:24.619478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:24.619439 5035 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 02:30:24.619719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:24.619495 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 02:30:25.619478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:25.619431 5035 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 02:30:25.620122 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:25.619502 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 02:30:26.670293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.670252 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:26.671455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.671400 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:26.672607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.672517 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:30:26.929998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.929892 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7"] Oct 30 02:30:26.931050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931020 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerName="cleanup" Oct 30 02:30:26.931050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931046 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931063 5035 cpu_manager.go:414] "containerMap: removing container" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931069 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931080 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931087 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931097 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931105 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931121 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931129 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931139 5035 cpu_manager.go:414] "containerMap: removing container" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931143 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931150 5035 cpu_manager.go:414] "containerMap: removing container" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931155 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931166 5035 cpu_manager.go:414] "containerMap: removing container" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931175 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931200 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931205 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931213 5035 cpu_manager.go:414] "containerMap: removing container" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931217 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931227 5035 cpu_manager.go:414] "containerMap: removing container" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931232 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931266 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931271 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931291 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931296 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerName="cleanup" Oct 30 02:30:26.931294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931310 5035 cpu_manager.go:414] "containerMap: removing container" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931314 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931470 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931479 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931486 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931494 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931500 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931507 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931513 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931520 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931544 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="103ec715-4f09-41c8-adba-c72457e4e9e1" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931558 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931568 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931580 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931588 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerName="cleanup" Oct 30 02:30:26.932590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.931595 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerName="cleanup" Oct 30 02:30:26.937738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.937713 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.957188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.957147 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-credential-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.957367 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.957287 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.957367 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.957322 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-certs\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.957511 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.957491 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-envoy\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.957641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.957623 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4167d08c-361b-4849-bdce-61e7650f13fa-istio-podinfo\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.957687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.957675 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2bg\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-kube-api-access-6h2bg\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.957753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.957722 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-istio-token\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.957787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.957770 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-data\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.957838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.957822 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4167d08c-361b-4849-bdce-61e7650f13fa-istiod-ca-cert\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:26.958293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.958266 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1fddb7-openshift-default-dockercfg-rwsrf\"" Oct 30 02:30:26.965598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:26.965561 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7"] Oct 30 02:30:27.059963 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.059914 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.059979 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-certs\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.060122 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-envoy\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.060182 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4167d08c-361b-4849-bdce-61e7650f13fa-istio-podinfo\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.060232 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2bg\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-kube-api-access-6h2bg\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.060270 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-istio-token\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.060312 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-data\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.060364 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4167d08c-361b-4849-bdce-61e7650f13fa-istiod-ca-cert\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.060384 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060734 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.060510 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-certs\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.060734 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.060608 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-credential-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.061165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.061121 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-data\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.061432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.061383 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-credential-socket\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.061806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.061778 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4167d08c-361b-4849-bdce-61e7650f13fa-istiod-ca-cert\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.063276 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.063251 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4167d08c-361b-4849-bdce-61e7650f13fa-istio-podinfo\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.063925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.063888 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-envoy\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.087223 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.087181 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2bg\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-kube-api-access-6h2bg\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.088660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.088120 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-istio-token\") pod \"e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.260648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.260606 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:27.619167 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:30:27.619135 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4167d08c_361b_4849_bdce_61e7650f13fa.slice/crio-a85fda227b52573cd486c3fb98d45a812cb1c5686e7609d84ca11fca4d4fcc44 WatchSource:0}: Error finding container a85fda227b52573cd486c3fb98d45a812cb1c5686e7609d84ca11fca4d4fcc44: Status 404 returned error can't find the container with id a85fda227b52573cd486c3fb98d45a812cb1c5686e7609d84ca11fca4d4fcc44 Oct 30 02:30:27.620561 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.620515 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7"] Oct 30 02:30:27.623169 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.623131 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:30:27.623272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.623210 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:30:27.623272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.623240 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:30:27.882095 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.881990 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" event={"ID":"4167d08c-361b-4849-bdce-61e7650f13fa","Type":"ContainerStarted","Data":"651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c"} Oct 30 02:30:27.882095 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.882069 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" event={"ID":"4167d08c-361b-4849-bdce-61e7650f13fa","Type":"ContainerStarted","Data":"a85fda227b52573cd486c3fb98d45a812cb1c5686e7609d84ca11fca4d4fcc44"} Oct 30 02:30:27.945473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:27.945415 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podStartSLOduration=1.945393986 podStartE2EDuration="1.945393986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:27.943189601 +0000 UTC m=+3706.446121833" watchObservedRunningTime="2026-10-30 02:30:27.945393986 +0000 UTC m=+3706.448326205" Oct 30 02:30:28.261051 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:28.261014 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:28.262634 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:28.262519 5035 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 02:30:28.262860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:28.262836 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 02:30:29.261135 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:29.261090 5035 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 02:30:29.261641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:29.261154 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 02:30:30.283550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:30.280395 5035 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 02:30:30.283550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:30.280503 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 02:30:31.261483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:31.261443 5035 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 02:30:31.261723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:31.261522 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 02:30:32.262108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:32.262065 5035 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 02:30:32.262629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:32.262253 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 02:30:33.261991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:33.261946 5035 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 02:30:33.262203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:33.262023 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 02:30:34.261504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:34.261455 5035 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 02:30:34.261717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:34.261557 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 02:30:35.267060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:35.267026 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:35.267516 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:35.267467 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:35.268209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:35.268191 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:30:47.633108 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:47.633066 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:30:50.546134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:50.546082 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp"] Oct 30 02:30:50.546788 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:50.546329 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" containerID="cri-o://82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3" gracePeriod=30 Oct 30 02:30:55.870462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:55.870429 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 02:30:56.015993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.015956 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-socket\") pod \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " Oct 30 02:30:56.016219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016004 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-data\") pod \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " Oct 30 02:30:56.016219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016108 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-envoy\") pod \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " Oct 30 02:30:56.016354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016231 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-credential-socket\") pod \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " Oct 30 02:30:56.016354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016279 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-podinfo\") pod \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " Oct 30 02:30:56.016354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016291 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-socket" pod "23e35d96-3162-49f8-bfa0-070f1f4da6d6" (UID: "23e35d96-3162-49f8-bfa0-070f1f4da6d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:56.016551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016362 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-data" pod "23e35d96-3162-49f8-bfa0-070f1f4da6d6" (UID: "23e35d96-3162-49f8-bfa0-070f1f4da6d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:56.016551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016417 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-token\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-token\") pod \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " Oct 30 02:30:56.016551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016497 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-kube-api-access-ntkpc\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-kube-api-access-ntkpc\") pod \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " Oct 30 02:30:56.016731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016584 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-credential-socket" pod "23e35d96-3162-49f8-bfa0-070f1f4da6d6" (UID: "23e35d96-3162-49f8-bfa0-070f1f4da6d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:56.016731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016605 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istiod-ca-cert\") pod \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " Oct 30 02:30:56.016731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.016643 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-certs\") pod \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\" (UID: \"23e35d96-3162-49f8-bfa0-070f1f4da6d6\") " Oct 30 02:30:56.017543 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.017497 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-certs" pod "23e35d96-3162-49f8-bfa0-070f1f4da6d6" (UID: "23e35d96-3162-49f8-bfa0-070f1f4da6d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:56.017709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.017552 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istiod-ca-cert" pod "23e35d96-3162-49f8-bfa0-070f1f4da6d6" (UID: "23e35d96-3162-49f8-bfa0-070f1f4da6d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:56.017709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.017660 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:56.017709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.017678 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:56.017709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.017689 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:56.019026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.018997 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-kube-api-access-ntkpc" pod "23e35d96-3162-49f8-bfa0-070f1f4da6d6" (UID: "23e35d96-3162-49f8-bfa0-070f1f4da6d6"). InnerVolumeSpecName "kube-api-access-ntkpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:56.019156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.019029 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-podinfo" pod "23e35d96-3162-49f8-bfa0-070f1f4da6d6" (UID: "23e35d96-3162-49f8-bfa0-070f1f4da6d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:30:56.019156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.019056 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-token" pod "23e35d96-3162-49f8-bfa0-070f1f4da6d6" (UID: "23e35d96-3162-49f8-bfa0-070f1f4da6d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:56.019156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.019088 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-envoy" pod "23e35d96-3162-49f8-bfa0-070f1f4da6d6" (UID: "23e35d96-3162-49f8-bfa0-070f1f4da6d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:56.119410 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.119312 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:56.119410 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.119347 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:56.119410 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.119357 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:56.119410 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.119367 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:56.119410 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.119375 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:56.119410 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.119384 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntkpc\" (UniqueName: \"kubernetes.io/projected/23e35d96-3162-49f8-bfa0-070f1f4da6d6-kube-api-access-ntkpc\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:30:56.323497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.323456 5035 generic.go:356] "Generic (PLEG): container finished" podID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerID="82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3" exitCode=0 Oct 30 02:30:56.323772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.323508 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" Oct 30 02:30:56.323772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.323549 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" event={"ID":"23e35d96-3162-49f8-bfa0-070f1f4da6d6","Type":"ContainerDied","Data":"82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3"} Oct 30 02:30:56.323772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.323611 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp" event={"ID":"23e35d96-3162-49f8-bfa0-070f1f4da6d6","Type":"ContainerDied","Data":"d4142ff86b15ea0ffa52d5bfb52c92d6a4cbe521ce1e9167d454638c6efc89eb"} Oct 30 02:30:56.323772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.323644 5035 scope.go:122] "RemoveContainer" containerID="82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3" Oct 30 02:30:56.375666 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.375626 5035 scope.go:122] "RemoveContainer" containerID="82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3" Oct 30 02:30:56.376214 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:30:56.376151 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3\": container with ID starting with 82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3 not found: ID does not exist" containerID="82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3" Oct 30 02:30:56.376214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.376199 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3"} err="failed to get container status \"82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3\": rpc error: code = NotFound desc = could not find container \"82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3\": container with ID starting with 82a0b8fee71c97a6124cb55a2fe6828615037529ef9fe613c2b2f39203157bb3 not found: ID does not exist" Oct 30 02:30:56.413856 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.413814 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp"] Oct 30 02:30:56.418691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:56.418657 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-4fmkp"] Oct 30 02:30:57.647513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:30:57.647477 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" path="/var/lib/kubelet/pods/23e35d96-3162-49f8-bfa0-070f1f4da6d6/volumes" Oct 30 02:31:17.631833 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:31:17.631788 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:31:47.632240 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:31:47.632201 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:32:17.631770 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:32:17.631726 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:32:47.631768 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:32:47.631724 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:33:17.631689 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:33:17.631644 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:33:26.906765 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:26.906725 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt"] Oct 30 02:33:31.908737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:31.908616 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" podUID="ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" containerName="payload-pre-processing" containerID="cri-o://150ad4d584d538fa4d8c786c8aa8bad6793a101c579c4ed9a81dd398d5b7cf02" gracePeriod=25 Oct 30 02:33:33.460968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:33.460919 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" podUID="ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:33:36.902085 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:36.902042 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7"] Oct 30 02:33:36.902626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:36.902333 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" podUID="f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" containerName="maas-api" containerID="cri-o://b793d62276258bba3a73b0f5587369e5cc2caaddab1cb0c3814fb3415ac16e86" gracePeriod=30 Oct 30 02:33:38.051340 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.051302 5035 generic.go:356] "Generic (PLEG): container finished" podID="f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" containerID="b793d62276258bba3a73b0f5587369e5cc2caaddab1cb0c3814fb3415ac16e86" exitCode=0 Oct 30 02:33:38.051838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.051369 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" event={"ID":"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a","Type":"ContainerDied","Data":"b793d62276258bba3a73b0f5587369e5cc2caaddab1cb0c3814fb3415ac16e86"} Oct 30 02:33:38.213761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.213681 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:33:38.286967 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.286929 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-openshift-service-ca\") pod \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " Oct 30 02:33:38.287253 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.287226 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-metrics-certs\") pod \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " Oct 30 02:33:38.287403 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.287335 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-openshift-service-ca" pod "f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" (UID: "f2365bd2-1955-46d8-ae7d-4cf401fe8a9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:33:38.287507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.287490 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-maas-api-tls\") pod \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " Oct 30 02:33:38.287629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.287605 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-kube-api-access-wr6wf\" (UniqueName: \"kubernetes.io/projected/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-kube-api-access-wr6wf\") pod \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\" (UID: \"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a\") " Oct 30 02:33:38.290025 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.289996 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:33:38.294164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.294129 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-kube-api-access-wr6wf" pod "f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" (UID: "f2365bd2-1955-46d8-ae7d-4cf401fe8a9a"). InnerVolumeSpecName "kube-api-access-wr6wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:38.298493 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.298456 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-metrics-certs" pod "f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" (UID: "f2365bd2-1955-46d8-ae7d-4cf401fe8a9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:33:38.308156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.308111 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-maas-api-tls" pod "f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" (UID: "f2365bd2-1955-46d8-ae7d-4cf401fe8a9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:33:38.392121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.392083 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:33:38.392121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.392116 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:33:38.392121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:38.392126 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr6wf\" (UniqueName: \"kubernetes.io/projected/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a-kube-api-access-wr6wf\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:33:39.062175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:39.062143 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" Oct 30 02:33:39.062175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:39.062158 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7" event={"ID":"f2365bd2-1955-46d8-ae7d-4cf401fe8a9a","Type":"ContainerDied","Data":"dd3c13db1dddc2a0e9abad0d7d88c29840e008278d930c228e9dfd61def6da1f"} Oct 30 02:33:39.062780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:39.062220 5035 scope.go:122] "RemoveContainer" containerID="b793d62276258bba3a73b0f5587369e5cc2caaddab1cb0c3814fb3415ac16e86" Oct 30 02:33:39.126814 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:39.126768 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7"] Oct 30 02:33:39.140946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:39.140903 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5da6-797f5b954-6l9g7"] Oct 30 02:33:39.648715 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:39.648674 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" path="/var/lib/kubelet/pods/f2365bd2-1955-46d8-ae7d-4cf401fe8a9a/volumes" Oct 30 02:33:43.461011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:43.460966 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" podUID="ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:33:44.011948 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:44.011915 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:33:44.022770 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:44.022735 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:33:44.027265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:44.027239 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:33:44.076461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:44.076433 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:33:44.087424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:44.087397 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:33:44.091994 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:44.091967 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:33:44.182293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:44.182262 5035 scope.go:122] "RemoveContainer" containerID="d71b9462e8bb4b99b6da72f6b827e4fd34da21daaa44e135116df63174c66423" Oct 30 02:33:47.634124 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:33:47.634082 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:33:49.710146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:49.710098 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2"] Oct 30 02:33:52.198079 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:52.197981 5035 generic.go:356] "Generic (PLEG): container finished" podID="ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" containerID="150ad4d584d538fa4d8c786c8aa8bad6793a101c579c4ed9a81dd398d5b7cf02" exitCode=0 Oct 30 02:33:52.198079 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:52.198072 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" event={"ID":"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06","Type":"ContainerDied","Data":"150ad4d584d538fa4d8c786c8aa8bad6793a101c579c4ed9a81dd398d5b7cf02"} Oct 30 02:33:52.256305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:52.256275 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:33:52.347491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:52.347425 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-kube-api-access-tcqdq\" (UniqueName: \"kubernetes.io/projected/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-kube-api-access-tcqdq\") pod \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\" (UID: \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\") " Oct 30 02:33:52.347758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:52.347617 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-plugins-config-volume\") pod \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\" (UID: \"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06\") " Oct 30 02:33:52.348141 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:52.348109 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-plugins-config-volume" pod "ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" (UID: "ceeb8f86-3769-42a2-8bcb-5a121ff0ec06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:33:52.348561 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:52.348509 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:33:52.350618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:52.350577 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-kube-api-access-tcqdq" pod "ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" (UID: "ceeb8f86-3769-42a2-8bcb-5a121ff0ec06"). InnerVolumeSpecName "kube-api-access-tcqdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:52.451048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:52.451007 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqdq\" (UniqueName: \"kubernetes.io/projected/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06-kube-api-access-tcqdq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:33:53.209007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:53.208968 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" Oct 30 02:33:53.209007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:53.208996 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt" event={"ID":"ceeb8f86-3769-42a2-8bcb-5a121ff0ec06","Type":"ContainerDied","Data":"e808e3e761d9f516878c6ba1f5d2b0c921b3059518c74a437f6b4e69b8695811"} Oct 30 02:33:53.209642 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:53.209068 5035 scope.go:122] "RemoveContainer" containerID="150ad4d584d538fa4d8c786c8aa8bad6793a101c579c4ed9a81dd398d5b7cf02" Oct 30 02:33:53.269199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:53.269162 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt"] Oct 30 02:33:53.282596 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:53.282554 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5da6-697744bdcf-cvtxt"] Oct 30 02:33:53.646012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:53.645925 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" path="/var/lib/kubelet/pods/ceeb8f86-3769-42a2-8bcb-5a121ff0ec06/volumes" Oct 30 02:33:54.712064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:54.712014 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" podUID="8e81adac-fa2a-4546-aed3-3682ced6fa8f" containerName="payload-processing" containerID="cri-o://8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7" gracePeriod=25 Oct 30 02:33:55.725775 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:33:55.725736 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" podUID="8e81adac-fa2a-4546-aed3-3682ced6fa8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:34:05.725023 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:05.724984 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" podUID="8e81adac-fa2a-4546-aed3-3682ced6fa8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:34:15.029178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.029142 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:34:15.117483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.117416 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e81adac-fa2a-4546-aed3-3682ced6fa8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81adac-fa2a-4546-aed3-3682ced6fa8f-plugins-config-volume\") pod \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\" (UID: \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\") " Oct 30 02:34:15.117758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.117632 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e81adac-fa2a-4546-aed3-3682ced6fa8f-kube-api-access-kz2hg\" (UniqueName: \"kubernetes.io/projected/8e81adac-fa2a-4546-aed3-3682ced6fa8f-kube-api-access-kz2hg\") pod \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\" (UID: \"8e81adac-fa2a-4546-aed3-3682ced6fa8f\") " Oct 30 02:34:15.117941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.117913 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e81adac-fa2a-4546-aed3-3682ced6fa8f-plugins-config-volume" pod "8e81adac-fa2a-4546-aed3-3682ced6fa8f" (UID: "8e81adac-fa2a-4546-aed3-3682ced6fa8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:15.118639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.118616 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e81adac-fa2a-4546-aed3-3682ced6fa8f-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:34:15.120253 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.120216 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e81adac-fa2a-4546-aed3-3682ced6fa8f-kube-api-access-kz2hg" pod "8e81adac-fa2a-4546-aed3-3682ced6fa8f" (UID: "8e81adac-fa2a-4546-aed3-3682ced6fa8f"). InnerVolumeSpecName "kube-api-access-kz2hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:15.220441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.220403 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2hg\" (UniqueName: \"kubernetes.io/projected/8e81adac-fa2a-4546-aed3-3682ced6fa8f-kube-api-access-kz2hg\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:34:15.447448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.447342 5035 generic.go:356] "Generic (PLEG): container finished" podID="8e81adac-fa2a-4546-aed3-3682ced6fa8f" containerID="8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7" exitCode=0 Oct 30 02:34:15.447448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.447398 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" Oct 30 02:34:15.447448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.447428 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" event={"ID":"8e81adac-fa2a-4546-aed3-3682ced6fa8f","Type":"ContainerDied","Data":"8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7"} Oct 30 02:34:15.447776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.447483 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2" event={"ID":"8e81adac-fa2a-4546-aed3-3682ced6fa8f","Type":"ContainerDied","Data":"0677f3c06138a4cb5eeb0bb513474ab151ee79a965ff7bcc49deabf33c382fd4"} Oct 30 02:34:15.447776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.447507 5035 scope.go:122] "RemoveContainer" containerID="8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7" Oct 30 02:34:15.459906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.459884 5035 scope.go:122] "RemoveContainer" containerID="8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7" Oct 30 02:34:15.460255 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:34:15.460232 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7\": container with ID starting with 8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7 not found: ID does not exist" containerID="8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7" Oct 30 02:34:15.460345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.460264 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7"} err="failed to get container status \"8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7\": rpc error: code = NotFound desc = could not find container \"8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7\": container with ID starting with 8cd624e90d7fab99266a71c3e9040218c9464996b5bb1c904baebaf8dfa51ca7 not found: ID does not exist" Oct 30 02:34:15.507505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.507462 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2"] Oct 30 02:34:15.520787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.520747 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d97e4c-7767d8db7d-wtkk2"] Oct 30 02:34:15.646509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:15.646470 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e81adac-fa2a-4546-aed3-3682ced6fa8f" path="/var/lib/kubelet/pods/8e81adac-fa2a-4546-aed3-3682ced6fa8f/volumes" Oct 30 02:34:17.632456 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:34:17.632414 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:34:42.651366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:42.651323 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p"] Oct 30 02:34:42.651986 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:42.651607 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" podUID="757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" containerName="maas-api" containerID="cri-o://6be532caf078b7039d3f5ee9adf83ffe291b380569ace64a86af03fdbd0a168c" gracePeriod=30 Oct 30 02:34:42.795085 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:42.795044 5035 generic.go:356] "Generic (PLEG): container finished" podID="757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" containerID="6be532caf078b7039d3f5ee9adf83ffe291b380569ace64a86af03fdbd0a168c" exitCode=0 Oct 30 02:34:42.795308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:42.795121 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" event={"ID":"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c","Type":"ContainerDied","Data":"6be532caf078b7039d3f5ee9adf83ffe291b380569ace64a86af03fdbd0a168c"} Oct 30 02:34:43.187550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.187214 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:34:43.322359 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.322316 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-kube-api-access-npks6\" (UniqueName: \"kubernetes.io/projected/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-kube-api-access-npks6\") pod \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " Oct 30 02:34:43.322654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.322387 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-metrics-certs\") pod \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " Oct 30 02:34:43.322654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.322589 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-openshift-service-ca\") pod \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " Oct 30 02:34:43.322791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.322676 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-maas-api-tls\") pod \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\" (UID: \"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c\") " Oct 30 02:34:43.336382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.330771 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-openshift-service-ca" pod "757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" (UID: "757b9410-ecd5-4cb2-88c3-5aec3f3eff8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:43.354639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.353556 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-kube-api-access-npks6" pod "757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" (UID: "757b9410-ecd5-4cb2-88c3-5aec3f3eff8c"). InnerVolumeSpecName "kube-api-access-npks6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:43.354639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.354479 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-metrics-certs" pod "757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" (UID: "757b9410-ecd5-4cb2-88c3-5aec3f3eff8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:43.364786 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.364732 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-maas-api-tls" pod "757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" (UID: "757b9410-ecd5-4cb2-88c3-5aec3f3eff8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:43.424708 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.424667 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npks6\" (UniqueName: \"kubernetes.io/projected/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-kube-api-access-npks6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:34:43.424708 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.424706 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:34:43.424708 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.424715 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:34:43.425097 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.424725 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:34:43.806955 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.806917 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" Oct 30 02:34:43.806955 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.806939 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p" event={"ID":"757b9410-ecd5-4cb2-88c3-5aec3f3eff8c","Type":"ContainerDied","Data":"ed9108bb6f483acfc64f0f54685a11b76315161e604a8d1910f41e2a18c91f28"} Oct 30 02:34:43.807470 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.807007 5035 scope.go:122] "RemoveContainer" containerID="6be532caf078b7039d3f5ee9adf83ffe291b380569ace64a86af03fdbd0a168c" Oct 30 02:34:43.872785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.872744 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p"] Oct 30 02:34:43.885348 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:43.885315 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d97e4c-74ffccd64b-72x6p"] Oct 30 02:34:45.660852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:45.660808 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" path="/var/lib/kubelet/pods/757b9410-ecd5-4cb2-88c3-5aec3f3eff8c/volumes" Oct 30 02:34:47.632219 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:34:47.632180 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:34:51.981910 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:51.981863 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j"] Oct 30 02:34:56.982476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:34:56.982402 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" podUID="4d8b92e3-f14b-4575-bdde-de31e8696afa" containerName="payload-pre-processing" containerID="cri-o://ed8edf3fc2438a744c33cf195862863b55ebf4183034868274a4e4515da81b49" gracePeriod=25 Oct 30 02:35:01.007782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:01.007734 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" podUID="4d8b92e3-f14b-4575-bdde-de31e8696afa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:35:11.016463 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:11.016421 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" podUID="4d8b92e3-f14b-4575-bdde-de31e8696afa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:35:17.164427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:17.164374 5035 generic.go:356] "Generic (PLEG): container finished" podID="4d8b92e3-f14b-4575-bdde-de31e8696afa" containerID="ed8edf3fc2438a744c33cf195862863b55ebf4183034868274a4e4515da81b49" exitCode=0 Oct 30 02:35:17.164850 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:17.164449 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" event={"ID":"4d8b92e3-f14b-4575-bdde-de31e8696afa","Type":"ContainerDied","Data":"ed8edf3fc2438a744c33cf195862863b55ebf4183034868274a4e4515da81b49"} Oct 30 02:35:17.286648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:17.286618 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:35:17.393863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:17.393821 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8b92e3-f14b-4575-bdde-de31e8696afa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8b92e3-f14b-4575-bdde-de31e8696afa-plugins-config-volume\") pod \"4d8b92e3-f14b-4575-bdde-de31e8696afa\" (UID: \"4d8b92e3-f14b-4575-bdde-de31e8696afa\") " Oct 30 02:35:17.394053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:17.393893 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8b92e3-f14b-4575-bdde-de31e8696afa-kube-api-access-2g4qw\" (UniqueName: \"kubernetes.io/projected/4d8b92e3-f14b-4575-bdde-de31e8696afa-kube-api-access-2g4qw\") pod \"4d8b92e3-f14b-4575-bdde-de31e8696afa\" (UID: \"4d8b92e3-f14b-4575-bdde-de31e8696afa\") " Oct 30 02:35:17.394303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:17.394277 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8b92e3-f14b-4575-bdde-de31e8696afa-plugins-config-volume" pod "4d8b92e3-f14b-4575-bdde-de31e8696afa" (UID: "4d8b92e3-f14b-4575-bdde-de31e8696afa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:17.394799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:17.394776 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8b92e3-f14b-4575-bdde-de31e8696afa-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:17.396329 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:17.396295 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8b92e3-f14b-4575-bdde-de31e8696afa-kube-api-access-2g4qw" pod "4d8b92e3-f14b-4575-bdde-de31e8696afa" (UID: "4d8b92e3-f14b-4575-bdde-de31e8696afa"). InnerVolumeSpecName "kube-api-access-2g4qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:17.496291 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:17.496185 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g4qw\" (UniqueName: \"kubernetes.io/projected/4d8b92e3-f14b-4575-bdde-de31e8696afa-kube-api-access-2g4qw\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:17.632377 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:35:17.632338 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:35:18.174684 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:18.174650 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" Oct 30 02:35:18.175212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:18.174687 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j" event={"ID":"4d8b92e3-f14b-4575-bdde-de31e8696afa","Type":"ContainerDied","Data":"076d3fd0a7bef001cd378d9ca88851611a012a81b1305120e684f335adf61790"} Oct 30 02:35:18.175212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:18.174758 5035 scope.go:122] "RemoveContainer" containerID="ed8edf3fc2438a744c33cf195862863b55ebf4183034868274a4e4515da81b49" Oct 30 02:35:18.247946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:18.247907 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j"] Oct 30 02:35:18.255153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:18.255113 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4593af-79c769d8bc-vmp5j"] Oct 30 02:35:18.696777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:18.696716 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7"] Oct 30 02:35:18.697087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:18.697040 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" containerID="cri-o://c09fa3242ce1a7a69f45561aff60ea94ad13b2ffa0adfe854e276ecea7554704" gracePeriod=30 Oct 30 02:35:19.681095 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:19.681053 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d8b92e3-f14b-4575-bdde-de31e8696afa" path="/var/lib/kubelet/pods/4d8b92e3-f14b-4575-bdde-de31e8696afa/volumes" Oct 30 02:35:24.233767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.233729 5035 generic.go:356] "Generic (PLEG): container finished" podID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerID="c09fa3242ce1a7a69f45561aff60ea94ad13b2ffa0adfe854e276ecea7554704" exitCode=0 Oct 30 02:35:24.234292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.233808 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" event={"ID":"29aa42b6-d6e4-417e-8e13-550d0e7020c7","Type":"ContainerDied","Data":"c09fa3242ce1a7a69f45561aff60ea94ad13b2ffa0adfe854e276ecea7554704"} Oct 30 02:35:24.429337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.429306 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:35:24.545607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.545214 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-envoy\") pod \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " Oct 30 02:35:24.545607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.545420 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-podinfo\") pod \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " Oct 30 02:35:24.545607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.545454 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-data\") pod \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " Oct 30 02:35:24.545607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.545552 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-token\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-token\") pod \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " Oct 30 02:35:24.545607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.545599 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-socket\") pod \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " Oct 30 02:35:24.546027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.545668 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istiod-ca-cert\") pod \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " Oct 30 02:35:24.546027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.545709 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-kube-api-access-sqcwf\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-kube-api-access-sqcwf\") pod \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " Oct 30 02:35:24.546027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.545869 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-certs\") pod \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " Oct 30 02:35:24.546027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.545967 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-credential-socket\") pod \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\" (UID: \"29aa42b6-d6e4-417e-8e13-550d0e7020c7\") " Oct 30 02:35:24.546615 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.546578 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-certs" pod "29aa42b6-d6e4-417e-8e13-550d0e7020c7" (UID: "29aa42b6-d6e4-417e-8e13-550d0e7020c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.546847 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.546822 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-data" pod "29aa42b6-d6e4-417e-8e13-550d0e7020c7" (UID: "29aa42b6-d6e4-417e-8e13-550d0e7020c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.547150 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.547121 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-socket" pod "29aa42b6-d6e4-417e-8e13-550d0e7020c7" (UID: "29aa42b6-d6e4-417e-8e13-550d0e7020c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.566626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.547309 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istiod-ca-cert" pod "29aa42b6-d6e4-417e-8e13-550d0e7020c7" (UID: "29aa42b6-d6e4-417e-8e13-550d0e7020c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:24.566626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.551626 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.566626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.551658 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.566626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.551672 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.566626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.551686 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.566626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.559603 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-credential-socket" pod "29aa42b6-d6e4-417e-8e13-550d0e7020c7" (UID: "29aa42b6-d6e4-417e-8e13-550d0e7020c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.608748 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.596277 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-kube-api-access-sqcwf" pod "29aa42b6-d6e4-417e-8e13-550d0e7020c7" (UID: "29aa42b6-d6e4-417e-8e13-550d0e7020c7"). InnerVolumeSpecName "kube-api-access-sqcwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:24.608748 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.603821 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-token" pod "29aa42b6-d6e4-417e-8e13-550d0e7020c7" (UID: "29aa42b6-d6e4-417e-8e13-550d0e7020c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:24.608748 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.608026 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-envoy" pod "29aa42b6-d6e4-417e-8e13-550d0e7020c7" (UID: "29aa42b6-d6e4-417e-8e13-550d0e7020c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:24.609220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.609191 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-podinfo" pod "29aa42b6-d6e4-417e-8e13-550d0e7020c7" (UID: "29aa42b6-d6e4-417e-8e13-550d0e7020c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:35:24.653643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.653608 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqcwf\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-kube-api-access-sqcwf\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.653643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.653639 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.653643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.653649 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.653902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.653658 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:24.653902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:24.653666 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29aa42b6-d6e4-417e-8e13-550d0e7020c7-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:25.244032 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:25.244001 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" Oct 30 02:35:25.244521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:25.244036 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7" event={"ID":"29aa42b6-d6e4-417e-8e13-550d0e7020c7","Type":"ContainerDied","Data":"753f6e06742659c858f05529b550bb1008e1d38d9552c644478fc6a463c63931"} Oct 30 02:35:25.244521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:25.244138 5035 scope.go:122] "RemoveContainer" containerID="c09fa3242ce1a7a69f45561aff60ea94ad13b2ffa0adfe854e276ecea7554704" Oct 30 02:35:25.319084 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:25.319039 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7"] Oct 30 02:35:25.332681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:25.332641 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2wbd7"] Oct 30 02:35:25.646254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:25.646165 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" path="/var/lib/kubelet/pods/29aa42b6-d6e4-417e-8e13-550d0e7020c7/volumes" Oct 30 02:35:27.114936 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:27.114892 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv"] Oct 30 02:35:27.150248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:27.150209 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt"] Oct 30 02:35:32.115965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.115914 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" podUID="8a3c62f3-e816-4358-b53e-6b9698c66531" containerName="payload-processing" containerID="cri-o://ec93caf82af20ffa1de85259a8a1c35c50c2004fdfae3dca0aefab2b3c17f8a7" gracePeriod=25 Oct 30 02:35:32.150700 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.150651 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" podUID="d89a119f-d530-4b93-b381-eb23753d22ee" containerName="payload-pre-processing" containerID="cri-o://e05c913fc8df4404d4fb354be1627ce00354ff05f8379defcc0b12aab429855e" gracePeriod=25 Oct 30 02:35:32.328172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.327754 5035 generic.go:356] "Generic (PLEG): container finished" podID="8a3c62f3-e816-4358-b53e-6b9698c66531" containerID="ec93caf82af20ffa1de85259a8a1c35c50c2004fdfae3dca0aefab2b3c17f8a7" exitCode=0 Oct 30 02:35:32.328172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.328084 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" event={"ID":"8a3c62f3-e816-4358-b53e-6b9698c66531","Type":"ContainerDied","Data":"ec93caf82af20ffa1de85259a8a1c35c50c2004fdfae3dca0aefab2b3c17f8a7"} Oct 30 02:35:32.422044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.422012 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:35:32.519926 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.519872 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3c62f3-e816-4358-b53e-6b9698c66531-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3c62f3-e816-4358-b53e-6b9698c66531-plugins-config-volume\") pod \"8a3c62f3-e816-4358-b53e-6b9698c66531\" (UID: \"8a3c62f3-e816-4358-b53e-6b9698c66531\") " Oct 30 02:35:32.520209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.520190 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3c62f3-e816-4358-b53e-6b9698c66531-kube-api-access-zmv5m\" (UniqueName: \"kubernetes.io/projected/8a3c62f3-e816-4358-b53e-6b9698c66531-kube-api-access-zmv5m\") pod \"8a3c62f3-e816-4358-b53e-6b9698c66531\" (UID: \"8a3c62f3-e816-4358-b53e-6b9698c66531\") " Oct 30 02:35:32.520410 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.520379 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3c62f3-e816-4358-b53e-6b9698c66531-plugins-config-volume" pod "8a3c62f3-e816-4358-b53e-6b9698c66531" (UID: "8a3c62f3-e816-4358-b53e-6b9698c66531"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:32.520922 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.520893 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3c62f3-e816-4358-b53e-6b9698c66531-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:32.524393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.524330 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3c62f3-e816-4358-b53e-6b9698c66531-kube-api-access-zmv5m" pod "8a3c62f3-e816-4358-b53e-6b9698c66531" (UID: "8a3c62f3-e816-4358-b53e-6b9698c66531"). InnerVolumeSpecName "kube-api-access-zmv5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:32.622212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:32.622164 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmv5m\" (UniqueName: \"kubernetes.io/projected/8a3c62f3-e816-4358-b53e-6b9698c66531-kube-api-access-zmv5m\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:33.337870 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:33.337832 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" Oct 30 02:35:33.337870 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:33.337854 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv" event={"ID":"8a3c62f3-e816-4358-b53e-6b9698c66531","Type":"ContainerDied","Data":"894a14506471acca25182954f1a092708b17f907ba3bcf0165154521b47787fb"} Oct 30 02:35:33.338430 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:33.337927 5035 scope.go:122] "RemoveContainer" containerID="ec93caf82af20ffa1de85259a8a1c35c50c2004fdfae3dca0aefab2b3c17f8a7" Oct 30 02:35:33.398111 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:33.398065 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv"] Oct 30 02:35:33.411758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:33.411718 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddb7-77f8d94d4d-vxwlv"] Oct 30 02:35:33.646549 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:33.646453 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3c62f3-e816-4358-b53e-6b9698c66531" path="/var/lib/kubelet/pods/8a3c62f3-e816-4358-b53e-6b9698c66531/volumes" Oct 30 02:35:34.814827 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:34.814784 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" podUID="d89a119f-d530-4b93-b381-eb23753d22ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:35:44.812946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:44.812901 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd"] Oct 30 02:35:44.813406 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:44.813172 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" podUID="7d492b6e-b398-48f6-b279-ef1ccd93f472" containerName="maas-api" containerID="cri-o://ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7" gracePeriod=30 Oct 30 02:35:44.814482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:44.814450 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" podUID="d89a119f-d530-4b93-b381-eb23753d22ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:35:45.326603 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.321993 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:35:45.447864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.447739 5035 generic.go:356] "Generic (PLEG): container finished" podID="7d492b6e-b398-48f6-b279-ef1ccd93f472" containerID="ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7" exitCode=0 Oct 30 02:35:45.447864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.447793 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" Oct 30 02:35:45.447864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.447824 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" event={"ID":"7d492b6e-b398-48f6-b279-ef1ccd93f472","Type":"ContainerDied","Data":"ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7"} Oct 30 02:35:45.448229 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.447902 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd" event={"ID":"7d492b6e-b398-48f6-b279-ef1ccd93f472","Type":"ContainerDied","Data":"f047a44bb33571d19cbb5981e934498c326a8f1adae4d34e2c18041fa959f2f1"} Oct 30 02:35:45.448229 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.447935 5035 scope.go:122] "RemoveContainer" containerID="ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7" Oct 30 02:35:45.457738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.457704 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-maas-api-tls\") pod \"7d492b6e-b398-48f6-b279-ef1ccd93f472\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " Oct 30 02:35:45.457923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.457850 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d492b6e-b398-48f6-b279-ef1ccd93f472-kube-api-access-8n495\" (UniqueName: \"kubernetes.io/projected/7d492b6e-b398-48f6-b279-ef1ccd93f472-kube-api-access-8n495\") pod \"7d492b6e-b398-48f6-b279-ef1ccd93f472\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " Oct 30 02:35:45.457923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.457914 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d492b6e-b398-48f6-b279-ef1ccd93f472-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d492b6e-b398-48f6-b279-ef1ccd93f472-openshift-service-ca\") pod \"7d492b6e-b398-48f6-b279-ef1ccd93f472\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " Oct 30 02:35:45.458106 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.457979 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-metrics-certs\") pod \"7d492b6e-b398-48f6-b279-ef1ccd93f472\" (UID: \"7d492b6e-b398-48f6-b279-ef1ccd93f472\") " Oct 30 02:35:45.458376 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.458346 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d492b6e-b398-48f6-b279-ef1ccd93f472-openshift-service-ca" pod "7d492b6e-b398-48f6-b279-ef1ccd93f472" (UID: "7d492b6e-b398-48f6-b279-ef1ccd93f472"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:45.458754 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.458733 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d492b6e-b398-48f6-b279-ef1ccd93f472-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:45.460099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.460066 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-metrics-certs" pod "7d492b6e-b398-48f6-b279-ef1ccd93f472" (UID: "7d492b6e-b398-48f6-b279-ef1ccd93f472"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:45.460220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.460161 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d492b6e-b398-48f6-b279-ef1ccd93f472-kube-api-access-8n495" pod "7d492b6e-b398-48f6-b279-ef1ccd93f472" (UID: "7d492b6e-b398-48f6-b279-ef1ccd93f472"). InnerVolumeSpecName "kube-api-access-8n495". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:45.460343 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.460319 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-maas-api-tls" pod "7d492b6e-b398-48f6-b279-ef1ccd93f472" (UID: "7d492b6e-b398-48f6-b279-ef1ccd93f472"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:45.461637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.461619 5035 scope.go:122] "RemoveContainer" containerID="ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7" Oct 30 02:35:45.462006 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:35:45.461986 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7\": container with ID starting with ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7 not found: ID does not exist" containerID="ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7" Oct 30 02:35:45.462067 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.462014 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7"} err="failed to get container status \"ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7\": rpc error: code = NotFound desc = could not find container \"ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7\": container with ID starting with ce2739a7f615fd02d42c16f0dff7e0cc36dd682629ae32b99bb5906333398ad7 not found: ID does not exist" Oct 30 02:35:45.560869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.560826 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n495\" (UniqueName: \"kubernetes.io/projected/7d492b6e-b398-48f6-b279-ef1ccd93f472-kube-api-access-8n495\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:45.560869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.560865 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:45.560869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.560875 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d492b6e-b398-48f6-b279-ef1ccd93f472-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:45.793833 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.793789 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd"] Oct 30 02:35:45.807302 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:45.807260 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4593af-5989484456-chspd"] Oct 30 02:35:47.632649 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:35:47.632607 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:35:47.645760 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:47.645727 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d492b6e-b398-48f6-b279-ef1ccd93f472" path="/var/lib/kubelet/pods/7d492b6e-b398-48f6-b279-ef1ccd93f472/volumes" Oct 30 02:35:48.825426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.825383 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm"] Oct 30 02:35:48.826292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826275 5035 cpu_manager.go:414] "containerMap: removing container" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerName="cleanup" Oct 30 02:35:48.826347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826293 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerName="cleanup" Oct 30 02:35:48.826347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826307 5035 cpu_manager.go:414] "containerMap: removing container" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" Oct 30 02:35:48.826347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826313 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" Oct 30 02:35:48.826347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826325 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerName="cleanup" Oct 30 02:35:48.826347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826330 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerName="cleanup" Oct 30 02:35:48.826347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826340 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" containerName="maas-api" Oct 30 02:35:48.826347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826345 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" containerName="maas-api" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826355 5035 cpu_manager.go:414] "containerMap: removing container" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerName="cleanup" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826361 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerName="cleanup" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826379 5035 cpu_manager.go:414] "containerMap: removing container" podUID="7d492b6e-b398-48f6-b279-ef1ccd93f472" containerName="maas-api" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826383 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d492b6e-b398-48f6-b279-ef1ccd93f472" containerName="maas-api" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826398 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8a3c62f3-e816-4358-b53e-6b9698c66531" containerName="payload-processing" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826403 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3c62f3-e816-4358-b53e-6b9698c66531" containerName="payload-processing" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826414 5035 cpu_manager.go:414] "containerMap: removing container" podUID="4d8b92e3-f14b-4575-bdde-de31e8696afa" containerName="payload-pre-processing" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826419 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8b92e3-f14b-4575-bdde-de31e8696afa" containerName="payload-pre-processing" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826427 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" containerName="payload-pre-processing" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826432 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" containerName="payload-pre-processing" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826442 5035 cpu_manager.go:414] "containerMap: removing container" podUID="757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" containerName="maas-api" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826446 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" containerName="maas-api" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826460 5035 cpu_manager.go:414] "containerMap: removing container" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826464 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826472 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerName="cleanup" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826477 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0834bdb9-90d6-447f-bc74-dba6cc5cdfa4" containerName="cleanup" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826485 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8e81adac-fa2a-4546-aed3-3682ced6fa8f" containerName="payload-processing" Oct 30 02:35:48.826595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826490 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81adac-fa2a-4546-aed3-3682ced6fa8f" containerName="payload-processing" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826628 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2a4698-c3d7-4af1-9903-76566281a8ec" containerName="cleanup" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826640 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d492b6e-b398-48f6-b279-ef1ccd93f472" containerName="maas-api" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826648 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3c62f3-e816-4358-b53e-6b9698c66531" containerName="payload-processing" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826657 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8b92e3-f14b-4575-bdde-de31e8696afa" containerName="payload-pre-processing" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826664 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceeb8f86-3769-42a2-8bcb-5a121ff0ec06" containerName="payload-pre-processing" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826669 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="757b9410-ecd5-4cb2-88c3-5aec3f3eff8c" containerName="maas-api" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826676 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="29aa42b6-d6e4-417e-8e13-550d0e7020c7" containerName="istio-proxy" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826684 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81adac-fa2a-4546-aed3-3682ced6fa8f" containerName="payload-processing" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826696 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="540dd1e7-ebaf-4d58-83a1-fc2334bbfa12" containerName="cleanup" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826701 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bb030d-d1b0-4485-8d30-3fbdfa293226" containerName="cleanup" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826708 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e35d96-3162-49f8-bfa0-070f1f4da6d6" containerName="istio-proxy" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826713 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88bbf10-8b58-4b5d-a7f1-2ea77c6c18da" containerName="cleanup" Oct 30 02:35:48.827717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.826719 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2365bd2-1955-46d8-ae7d-4cf401fe8a9a" containerName="maas-api" Oct 30 02:35:48.830790 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.830769 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:48.874505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.874465 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm"] Oct 30 02:35:48.920396 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.920343 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:48.920596 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.920425 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:48.920596 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.920470 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:48.920763 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.920727 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:48.920822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.920809 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6b6p\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-kube-api-access-q6b6p\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:48.920882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.920838 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:48.921015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.920988 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:48.921096 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.921075 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:48.921148 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:48.921104 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.022697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.022648 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6b6p\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-kube-api-access-q6b6p\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.022962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.022718 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.022962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.022774 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.022962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.022841 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.022962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.022867 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.022962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.022947 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.023226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.022991 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.023226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.023038 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.023334 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.023244 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.023334 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.023248 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.023334 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.023306 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.023500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.023406 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.023677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.023649 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.024008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.023983 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.025433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.025407 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.025719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.025699 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.050701 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.050660 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.050956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.050927 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6b6p\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-kube-api-access-q6b6p\") pod \"maas-default-gateway-openshift-default-75d7c785f-5htrm\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.147707 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.147615 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:49.531347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.531316 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm"] Oct 30 02:35:49.531755 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:35:49.531729 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26c7197_b43a_4196_a5c3_4cbb88c5754f.slice/crio-91ea7651d24297e1347fa90a8f7a005630ea7e4a45dde444add335126552fd51 WatchSource:0}: Error finding container 91ea7651d24297e1347fa90a8f7a005630ea7e4a45dde444add335126552fd51: Status 404 returned error can't find the container with id 91ea7651d24297e1347fa90a8f7a005630ea7e4a45dde444add335126552fd51 Oct 30 02:35:49.535080 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.535046 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:35:49.535137 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.535121 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:35:49.535174 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:49.535148 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:35:50.244163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:50.244122 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf"] Oct 30 02:35:50.245039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:50.244345 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" containerID="cri-o://dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80" gracePeriod=30 Oct 30 02:35:50.547613 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:50.547339 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" event={"ID":"f26c7197-b43a-4196-a5c3-4cbb88c5754f","Type":"ContainerStarted","Data":"38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487"} Oct 30 02:35:50.547613 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:50.547419 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" event={"ID":"f26c7197-b43a-4196-a5c3-4cbb88c5754f","Type":"ContainerStarted","Data":"91ea7651d24297e1347fa90a8f7a005630ea7e4a45dde444add335126552fd51"} Oct 30 02:35:50.599896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:50.599822 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" podStartSLOduration=2.5998009140000002 podStartE2EDuration="2.599800914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:35:50.597904314 +0000 UTC m=+4029.100836534" watchObservedRunningTime="2026-10-30 02:35:50.599800914 +0000 UTC m=+4029.102733211" Oct 30 02:35:51.148280 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:51.148235 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:51.150359 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:51.150322 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5htrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:35:51.150522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:51.150393 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 02:35:51.904782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:51.903752 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj"] Oct 30 02:35:51.904782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:51.904030 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" podUID="12564f35-a37b-4dd7-befd-91ee03e6b39d" containerName="maas-api" containerID="cri-o://1e82963a423512d8876a86feb58a1336a57658ba0efbf098cf49e398a3406659" gracePeriod=30 Oct 30 02:35:52.205946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.185220 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5htrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:35:52.205946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.185280 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 02:35:52.605785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.605750 5035 generic.go:356] "Generic (PLEG): container finished" podID="12564f35-a37b-4dd7-befd-91ee03e6b39d" containerID="1e82963a423512d8876a86feb58a1336a57658ba0efbf098cf49e398a3406659" exitCode=0 Oct 30 02:35:52.606002 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.605976 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" event={"ID":"12564f35-a37b-4dd7-befd-91ee03e6b39d","Type":"ContainerDied","Data":"1e82963a423512d8876a86feb58a1336a57658ba0efbf098cf49e398a3406659"} Oct 30 02:35:52.614519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.606045 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" event={"ID":"12564f35-a37b-4dd7-befd-91ee03e6b39d","Type":"ContainerDied","Data":"3713843187f0b69e5f9a24e7fc43647bb513349c45ce955afba35e0d64a1a5ae"} Oct 30 02:35:52.614519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.606071 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3713843187f0b69e5f9a24e7fc43647bb513349c45ce955afba35e0d64a1a5ae" Oct 30 02:35:52.614519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.609505 5035 generic.go:356] "Generic (PLEG): container finished" podID="d89a119f-d530-4b93-b381-eb23753d22ee" containerID="e05c913fc8df4404d4fb354be1627ce00354ff05f8379defcc0b12aab429855e" exitCode=0 Oct 30 02:35:52.614519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.609560 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" event={"ID":"d89a119f-d530-4b93-b381-eb23753d22ee","Type":"ContainerDied","Data":"e05c913fc8df4404d4fb354be1627ce00354ff05f8379defcc0b12aab429855e"} Oct 30 02:35:52.614519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.609932 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:35:52.623049 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.623001 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12564f35-a37b-4dd7-befd-91ee03e6b39d-kube-api-access-m4lk6\" (UniqueName: \"kubernetes.io/projected/12564f35-a37b-4dd7-befd-91ee03e6b39d-kube-api-access-m4lk6\") pod \"12564f35-a37b-4dd7-befd-91ee03e6b39d\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " Oct 30 02:35:52.623300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.623150 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-metrics-certs\") pod \"12564f35-a37b-4dd7-befd-91ee03e6b39d\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " Oct 30 02:35:52.623373 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.623309 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-maas-api-tls\") pod \"12564f35-a37b-4dd7-befd-91ee03e6b39d\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " Oct 30 02:35:52.623373 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.623359 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12564f35-a37b-4dd7-befd-91ee03e6b39d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12564f35-a37b-4dd7-befd-91ee03e6b39d-openshift-service-ca\") pod \"12564f35-a37b-4dd7-befd-91ee03e6b39d\" (UID: \"12564f35-a37b-4dd7-befd-91ee03e6b39d\") " Oct 30 02:35:52.623964 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.623917 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12564f35-a37b-4dd7-befd-91ee03e6b39d-openshift-service-ca" pod "12564f35-a37b-4dd7-befd-91ee03e6b39d" (UID: "12564f35-a37b-4dd7-befd-91ee03e6b39d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:52.624509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.624481 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12564f35-a37b-4dd7-befd-91ee03e6b39d-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:52.626373 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.626308 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12564f35-a37b-4dd7-befd-91ee03e6b39d-kube-api-access-m4lk6" pod "12564f35-a37b-4dd7-befd-91ee03e6b39d" (UID: "12564f35-a37b-4dd7-befd-91ee03e6b39d"). InnerVolumeSpecName "kube-api-access-m4lk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:52.626752 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.626730 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-metrics-certs" pod "12564f35-a37b-4dd7-befd-91ee03e6b39d" (UID: "12564f35-a37b-4dd7-befd-91ee03e6b39d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:52.628484 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.628448 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-maas-api-tls" pod "12564f35-a37b-4dd7-befd-91ee03e6b39d" (UID: "12564f35-a37b-4dd7-befd-91ee03e6b39d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:52.720499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.720469 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:35:52.725997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.725962 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4lk6\" (UniqueName: \"kubernetes.io/projected/12564f35-a37b-4dd7-befd-91ee03e6b39d-kube-api-access-m4lk6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:52.725997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.725995 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:52.726213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.726007 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12564f35-a37b-4dd7-befd-91ee03e6b39d-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:52.933138 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.933041 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89a119f-d530-4b93-b381-eb23753d22ee-kube-api-access-r87gm\" (UniqueName: \"kubernetes.io/projected/d89a119f-d530-4b93-b381-eb23753d22ee-kube-api-access-r87gm\") pod \"d89a119f-d530-4b93-b381-eb23753d22ee\" (UID: \"d89a119f-d530-4b93-b381-eb23753d22ee\") " Oct 30 02:35:52.933606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.933284 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89a119f-d530-4b93-b381-eb23753d22ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89a119f-d530-4b93-b381-eb23753d22ee-plugins-config-volume\") pod \"d89a119f-d530-4b93-b381-eb23753d22ee\" (UID: \"d89a119f-d530-4b93-b381-eb23753d22ee\") " Oct 30 02:35:52.933708 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.933680 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89a119f-d530-4b93-b381-eb23753d22ee-plugins-config-volume" pod "d89a119f-d530-4b93-b381-eb23753d22ee" (UID: "d89a119f-d530-4b93-b381-eb23753d22ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:52.934124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.934102 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89a119f-d530-4b93-b381-eb23753d22ee-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:52.935440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:52.935414 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89a119f-d530-4b93-b381-eb23753d22ee-kube-api-access-r87gm" pod "d89a119f-d530-4b93-b381-eb23753d22ee" (UID: "d89a119f-d530-4b93-b381-eb23753d22ee"). InnerVolumeSpecName "kube-api-access-r87gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:53.035312 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.035273 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r87gm\" (UniqueName: \"kubernetes.io/projected/d89a119f-d530-4b93-b381-eb23753d22ee-kube-api-access-r87gm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:53.149698 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.149653 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5htrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:35:53.149897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.149717 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 02:35:53.619792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.619757 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" Oct 30 02:35:53.619792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.619780 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt" event={"ID":"d89a119f-d530-4b93-b381-eb23753d22ee","Type":"ContainerDied","Data":"cd7eb76b440820a5874ef314906915e06465692fab9632e789b157654bc4a27e"} Oct 30 02:35:53.620093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.619845 5035 scope.go:122] "RemoveContainer" containerID="e05c913fc8df4404d4fb354be1627ce00354ff05f8379defcc0b12aab429855e" Oct 30 02:35:53.620093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.619971 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj" Oct 30 02:35:53.720190 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.720150 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt"] Oct 30 02:35:53.727562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.727515 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddb7-df6d869cf-dr8jt"] Oct 30 02:35:53.764623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.764468 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj"] Oct 30 02:35:53.794679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:53.794624 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddb7-775d586875-mmbcj"] Oct 30 02:35:54.148411 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:54.148369 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5htrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:35:54.148991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:54.148441 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 02:35:55.148553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.148496 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5htrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:35:55.149031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.148637 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 02:35:55.619958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.619905 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:35:55.641883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.641840 5035 generic.go:356] "Generic (PLEG): container finished" podID="0175afca-7f43-471c-9ffb-bd96f1150806" containerID="dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80" exitCode=0 Oct 30 02:35:55.642113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.641965 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" Oct 30 02:35:55.647636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.647604 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12564f35-a37b-4dd7-befd-91ee03e6b39d" path="/var/lib/kubelet/pods/12564f35-a37b-4dd7-befd-91ee03e6b39d/volumes" Oct 30 02:35:55.648221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.648205 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89a119f-d530-4b93-b381-eb23753d22ee" path="/var/lib/kubelet/pods/d89a119f-d530-4b93-b381-eb23753d22ee/volumes" Oct 30 02:35:55.650665 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.650644 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" event={"ID":"0175afca-7f43-471c-9ffb-bd96f1150806","Type":"ContainerDied","Data":"dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80"} Oct 30 02:35:55.650737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.650700 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf" event={"ID":"0175afca-7f43-471c-9ffb-bd96f1150806","Type":"ContainerDied","Data":"6d32a31780e93623a401a9177a0cc6eb2f4ea0d060d9db7dd4d7d823d1b57f1b"} Oct 30 02:35:55.650737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.650727 5035 scope.go:122] "RemoveContainer" containerID="dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80" Oct 30 02:35:55.663820 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.663787 5035 scope.go:122] "RemoveContainer" containerID="dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80" Oct 30 02:35:55.664235 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:35:55.664208 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80\": container with ID starting with dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80 not found: ID does not exist" containerID="dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80" Oct 30 02:35:55.664317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.664248 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80"} err="failed to get container status \"dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80\": rpc error: code = NotFound desc = could not find container \"dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80\": container with ID starting with dc103b97de03f4b1899b30eb139ab31bbc1211f16a1bacf372a2e6ab5672cb80 not found: ID does not exist" Oct 30 02:35:55.813616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.813460 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0175afca-7f43-471c-9ffb-bd96f1150806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0175afca-7f43-471c-9ffb-bd96f1150806-istiod-ca-cert\") pod \"0175afca-7f43-471c-9ffb-bd96f1150806\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " Oct 30 02:35:55.813616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.813523 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-kube-api-access-jhpv8\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-kube-api-access-jhpv8\") pod \"0175afca-7f43-471c-9ffb-bd96f1150806\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " Oct 30 02:35:55.813616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.813615 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-data\") pod \"0175afca-7f43-471c-9ffb-bd96f1150806\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " Oct 30 02:35:55.813903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.813671 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-certs\") pod \"0175afca-7f43-471c-9ffb-bd96f1150806\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " Oct 30 02:35:55.813903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.813734 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-socket\") pod \"0175afca-7f43-471c-9ffb-bd96f1150806\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " Oct 30 02:35:55.814184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.813888 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-envoy\") pod \"0175afca-7f43-471c-9ffb-bd96f1150806\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " Oct 30 02:35:55.814432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.814059 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0175afca-7f43-471c-9ffb-bd96f1150806-istiod-ca-cert" pod "0175afca-7f43-471c-9ffb-bd96f1150806" (UID: "0175afca-7f43-471c-9ffb-bd96f1150806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:55.814551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.814149 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-socket" pod "0175afca-7f43-471c-9ffb-bd96f1150806" (UID: "0175afca-7f43-471c-9ffb-bd96f1150806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.814551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.814379 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-data" pod "0175afca-7f43-471c-9ffb-bd96f1150806" (UID: "0175afca-7f43-471c-9ffb-bd96f1150806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.814551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.814459 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0175afca-7f43-471c-9ffb-bd96f1150806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0175afca-7f43-471c-9ffb-bd96f1150806-istio-podinfo\") pod \"0175afca-7f43-471c-9ffb-bd96f1150806\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " Oct 30 02:35:55.814551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.814476 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-certs" pod "0175afca-7f43-471c-9ffb-bd96f1150806" (UID: "0175afca-7f43-471c-9ffb-bd96f1150806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.814795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.814550 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-istio-token\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-istio-token\") pod \"0175afca-7f43-471c-9ffb-bd96f1150806\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " Oct 30 02:35:55.814795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.814615 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-credential-socket\") pod \"0175afca-7f43-471c-9ffb-bd96f1150806\" (UID: \"0175afca-7f43-471c-9ffb-bd96f1150806\") " Oct 30 02:35:55.815045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.815017 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-credential-socket" pod "0175afca-7f43-471c-9ffb-bd96f1150806" (UID: "0175afca-7f43-471c-9ffb-bd96f1150806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.815646 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.815622 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.815753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.815647 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0175afca-7f43-471c-9ffb-bd96f1150806-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.815753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.815661 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.815753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.815681 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.815753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.815693 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.817101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.817077 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-kube-api-access-jhpv8" pod "0175afca-7f43-471c-9ffb-bd96f1150806" (UID: "0175afca-7f43-471c-9ffb-bd96f1150806"). InnerVolumeSpecName "kube-api-access-jhpv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:55.817346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.817310 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-envoy" pod "0175afca-7f43-471c-9ffb-bd96f1150806" (UID: "0175afca-7f43-471c-9ffb-bd96f1150806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.817473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.817453 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-istio-token" pod "0175afca-7f43-471c-9ffb-bd96f1150806" (UID: "0175afca-7f43-471c-9ffb-bd96f1150806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:55.817723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.817704 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0175afca-7f43-471c-9ffb-bd96f1150806-istio-podinfo" pod "0175afca-7f43-471c-9ffb-bd96f1150806" (UID: "0175afca-7f43-471c-9ffb-bd96f1150806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:35:55.916947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.916907 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0175afca-7f43-471c-9ffb-bd96f1150806-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.916947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.916943 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0175afca-7f43-471c-9ffb-bd96f1150806-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.916947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.916956 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.917226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:55.916965 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhpv8\" (UniqueName: \"kubernetes.io/projected/0175afca-7f43-471c-9ffb-bd96f1150806-kube-api-access-jhpv8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:35:56.019868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:56.019675 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf"] Oct 30 02:35:56.025519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:56.025479 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-mcvhf"] Oct 30 02:35:56.148495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:56.148396 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5htrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:35:56.148495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:56.148470 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 02:35:57.148295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:57.148248 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5htrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:35:57.148541 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:57.148337 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 02:35:57.649913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:57.649865 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" path="/var/lib/kubelet/pods/0175afca-7f43-471c-9ffb-bd96f1150806/volumes" Oct 30 02:35:58.768837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:58.768803 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:58.769419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:58.769394 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:35:58.770499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:35:58.770475 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:36:02.089087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:02.089043 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7"] Oct 30 02:36:02.089652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:02.089420 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" containerID="cri-o://651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c" gracePeriod=30 Oct 30 02:36:05.270838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:05.270792 5035 patch_prober.go:29] interesting pod/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:36:05.271273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:05.270884 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:36:07.388918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.388888 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:36:07.556389 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556344 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4167d08c-361b-4849-bdce-61e7650f13fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4167d08c-361b-4849-bdce-61e7650f13fa-istiod-ca-cert\") pod \"4167d08c-361b-4849-bdce-61e7650f13fa\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " Oct 30 02:36:07.556683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556450 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-credential-socket\") pod \"4167d08c-361b-4849-bdce-61e7650f13fa\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " Oct 30 02:36:07.556683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556476 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-socket\") pod \"4167d08c-361b-4849-bdce-61e7650f13fa\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " Oct 30 02:36:07.556683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556577 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-data\") pod \"4167d08c-361b-4849-bdce-61e7650f13fa\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " Oct 30 02:36:07.556683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556635 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-envoy\") pod \"4167d08c-361b-4849-bdce-61e7650f13fa\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " Oct 30 02:36:07.556929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556709 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-kube-api-access-6h2bg\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-kube-api-access-6h2bg\") pod \"4167d08c-361b-4849-bdce-61e7650f13fa\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " Oct 30 02:36:07.556929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556802 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4167d08c-361b-4849-bdce-61e7650f13fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4167d08c-361b-4849-bdce-61e7650f13fa-istio-podinfo\") pod \"4167d08c-361b-4849-bdce-61e7650f13fa\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " Oct 30 02:36:07.556929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556879 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-socket" pod "4167d08c-361b-4849-bdce-61e7650f13fa" (UID: "4167d08c-361b-4849-bdce-61e7650f13fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.556929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556892 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-credential-socket" pod "4167d08c-361b-4849-bdce-61e7650f13fa" (UID: "4167d08c-361b-4849-bdce-61e7650f13fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.556929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556905 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4167d08c-361b-4849-bdce-61e7650f13fa-istiod-ca-cert" pod "4167d08c-361b-4849-bdce-61e7650f13fa" (UID: "4167d08c-361b-4849-bdce-61e7650f13fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:07.556929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556919 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-istio-token\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-istio-token\") pod \"4167d08c-361b-4849-bdce-61e7650f13fa\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " Oct 30 02:36:07.557286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556929 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-data" pod "4167d08c-361b-4849-bdce-61e7650f13fa" (UID: "4167d08c-361b-4849-bdce-61e7650f13fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.557286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.556982 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-certs\") pod \"4167d08c-361b-4849-bdce-61e7650f13fa\" (UID: \"4167d08c-361b-4849-bdce-61e7650f13fa\") " Oct 30 02:36:07.557286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.557267 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-certs" pod "4167d08c-361b-4849-bdce-61e7650f13fa" (UID: "4167d08c-361b-4849-bdce-61e7650f13fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.557842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.557811 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4167d08c-361b-4849-bdce-61e7650f13fa-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.557983 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.557845 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.557983 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.557864 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.557983 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.557880 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.557983 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.557893 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.559181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.559149 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4167d08c-361b-4849-bdce-61e7650f13fa-istio-podinfo" pod "4167d08c-361b-4849-bdce-61e7650f13fa" (UID: "4167d08c-361b-4849-bdce-61e7650f13fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:36:07.559300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.559189 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-envoy" pod "4167d08c-361b-4849-bdce-61e7650f13fa" (UID: "4167d08c-361b-4849-bdce-61e7650f13fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:07.559300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.559208 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-kube-api-access-6h2bg" pod "4167d08c-361b-4849-bdce-61e7650f13fa" (UID: "4167d08c-361b-4849-bdce-61e7650f13fa"). InnerVolumeSpecName "kube-api-access-6h2bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:07.559300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.559291 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-istio-token" pod "4167d08c-361b-4849-bdce-61e7650f13fa" (UID: "4167d08c-361b-4849-bdce-61e7650f13fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:07.659521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.659484 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2bg\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-kube-api-access-6h2bg\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.659521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.659520 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4167d08c-361b-4849-bdce-61e7650f13fa-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.659521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.659542 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4167d08c-361b-4849-bdce-61e7650f13fa-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.659521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.659551 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4167d08c-361b-4849-bdce-61e7650f13fa-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:07.847396 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.847355 5035 generic.go:356] "Generic (PLEG): container finished" podID="4167d08c-361b-4849-bdce-61e7650f13fa" containerID="651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c" exitCode=0 Oct 30 02:36:07.847657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.847413 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" Oct 30 02:36:07.847657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.847443 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" event={"ID":"4167d08c-361b-4849-bdce-61e7650f13fa","Type":"ContainerDied","Data":"651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c"} Oct 30 02:36:07.847657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.847505 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7" event={"ID":"4167d08c-361b-4849-bdce-61e7650f13fa","Type":"ContainerDied","Data":"a85fda227b52573cd486c3fb98d45a812cb1c5686e7609d84ca11fca4d4fcc44"} Oct 30 02:36:07.847657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.847551 5035 scope.go:122] "RemoveContainer" containerID="651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c" Oct 30 02:36:07.859645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.859621 5035 scope.go:122] "RemoveContainer" containerID="651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c" Oct 30 02:36:07.860040 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:36:07.860016 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c\": container with ID starting with 651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c not found: ID does not exist" containerID="651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c" Oct 30 02:36:07.860123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.860055 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c"} err="failed to get container status \"651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c\": rpc error: code = NotFound desc = could not find container \"651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c\": container with ID starting with 651929f3a9913801c389439c4018da9cb0b3473ffff956595c9087458a15161c not found: ID does not exist" Oct 30 02:36:07.907190 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.907086 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7"] Oct 30 02:36:07.921152 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:07.921109 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddb7-openshift-default-85fb47fcf-7d6z7"] Oct 30 02:36:09.646419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:09.646381 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" path="/var/lib/kubelet/pods/4167d08c-361b-4849-bdce-61e7650f13fa/volumes" Oct 30 02:36:10.220852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:10.220807 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf"] Oct 30 02:36:15.222058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:15.221973 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" podUID="3bf1dda1-3b78-4da2-906b-8882ffee8025" containerName="payload-pre-processing" containerID="cri-o://7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37" gracePeriod=25 Oct 30 02:36:17.153728 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:17.153685 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" podUID="3bf1dda1-3b78-4da2-906b-8882ffee8025" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:36:17.630937 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:36:17.630893 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:36:27.153092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:27.153045 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" podUID="3bf1dda1-3b78-4da2-906b-8882ffee8025" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:36:28.331175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:28.331134 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs"] Oct 30 02:36:33.332597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:33.332515 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" podUID="f1fd086c-87ea-4a37-8ab3-d909e528c75a" containerName="payload-processing" containerID="cri-o://2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d" gracePeriod=25 Oct 30 02:36:33.643565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:33.643512 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:36:33.658715 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:33.658660 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fd086c-87ea-4a37-8ab3-d909e528c75a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fd086c-87ea-4a37-8ab3-d909e528c75a-plugins-config-volume\") pod \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\" (UID: \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\") " Oct 30 02:36:33.659040 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:33.658883 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fd086c-87ea-4a37-8ab3-d909e528c75a-kube-api-access-kl66f\" (UniqueName: \"kubernetes.io/projected/f1fd086c-87ea-4a37-8ab3-d909e528c75a-kube-api-access-kl66f\") pod \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\" (UID: \"f1fd086c-87ea-4a37-8ab3-d909e528c75a\") " Oct 30 02:36:33.659387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:33.659354 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fd086c-87ea-4a37-8ab3-d909e528c75a-plugins-config-volume" pod "f1fd086c-87ea-4a37-8ab3-d909e528c75a" (UID: "f1fd086c-87ea-4a37-8ab3-d909e528c75a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:33.659801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:33.659773 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fd086c-87ea-4a37-8ab3-d909e528c75a-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:33.662006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:33.661974 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fd086c-87ea-4a37-8ab3-d909e528c75a-kube-api-access-kl66f" pod "f1fd086c-87ea-4a37-8ab3-d909e528c75a" (UID: "f1fd086c-87ea-4a37-8ab3-d909e528c75a"). InnerVolumeSpecName "kube-api-access-kl66f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:33.761249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:33.761203 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl66f\" (UniqueName: \"kubernetes.io/projected/f1fd086c-87ea-4a37-8ab3-d909e528c75a-kube-api-access-kl66f\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:34.146717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:34.146679 5035 generic.go:356] "Generic (PLEG): container finished" podID="f1fd086c-87ea-4a37-8ab3-d909e528c75a" containerID="2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d" exitCode=0 Oct 30 02:36:34.146929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:34.146740 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" Oct 30 02:36:34.146929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:34.146763 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" event={"ID":"f1fd086c-87ea-4a37-8ab3-d909e528c75a","Type":"ContainerDied","Data":"2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d"} Oct 30 02:36:34.146929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:34.146857 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs" event={"ID":"f1fd086c-87ea-4a37-8ab3-d909e528c75a","Type":"ContainerDied","Data":"82d9c9f5a152b01ccab504cfeaf81c4d63cd98160c7960dd556f876837f1bddf"} Oct 30 02:36:34.146929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:34.146887 5035 scope.go:122] "RemoveContainer" containerID="2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d" Oct 30 02:36:34.159237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:34.159209 5035 scope.go:122] "RemoveContainer" containerID="2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d" Oct 30 02:36:34.159594 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:36:34.159565 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d\": container with ID starting with 2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d not found: ID does not exist" containerID="2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d" Oct 30 02:36:34.159697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:34.159605 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d"} err="failed to get container status \"2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d\": rpc error: code = NotFound desc = could not find container \"2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d\": container with ID starting with 2f4912528fd9c97f01de675640d0f73140f851fef6f10e6b13fdbca56fc8f65d not found: ID does not exist" Oct 30 02:36:34.211090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:34.211045 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs"] Oct 30 02:36:34.223864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:34.223823 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d207e8-597bf6d8c4-trvfs"] Oct 30 02:36:35.544602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:35.544572 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:36:35.588970 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:35.588922 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf1dda1-3b78-4da2-906b-8882ffee8025-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf1dda1-3b78-4da2-906b-8882ffee8025-plugins-config-volume\") pod \"3bf1dda1-3b78-4da2-906b-8882ffee8025\" (UID: \"3bf1dda1-3b78-4da2-906b-8882ffee8025\") " Oct 30 02:36:35.589195 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:35.589019 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf1dda1-3b78-4da2-906b-8882ffee8025-kube-api-access-pxwr5\" (UniqueName: \"kubernetes.io/projected/3bf1dda1-3b78-4da2-906b-8882ffee8025-kube-api-access-pxwr5\") pod \"3bf1dda1-3b78-4da2-906b-8882ffee8025\" (UID: \"3bf1dda1-3b78-4da2-906b-8882ffee8025\") " Oct 30 02:36:35.589600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:35.589518 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf1dda1-3b78-4da2-906b-8882ffee8025-plugins-config-volume" pod "3bf1dda1-3b78-4da2-906b-8882ffee8025" (UID: "3bf1dda1-3b78-4da2-906b-8882ffee8025"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:35.590064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:35.590027 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf1dda1-3b78-4da2-906b-8882ffee8025-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:35.594354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:35.594322 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf1dda1-3b78-4da2-906b-8882ffee8025-kube-api-access-pxwr5" pod "3bf1dda1-3b78-4da2-906b-8882ffee8025" (UID: "3bf1dda1-3b78-4da2-906b-8882ffee8025"). InnerVolumeSpecName "kube-api-access-pxwr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:35.658101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:35.658061 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fd086c-87ea-4a37-8ab3-d909e528c75a" path="/var/lib/kubelet/pods/f1fd086c-87ea-4a37-8ab3-d909e528c75a/volumes" Oct 30 02:36:35.691668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:35.691627 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxwr5\" (UniqueName: \"kubernetes.io/projected/3bf1dda1-3b78-4da2-906b-8882ffee8025-kube-api-access-pxwr5\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:36.168766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:36.168728 5035 generic.go:356] "Generic (PLEG): container finished" podID="3bf1dda1-3b78-4da2-906b-8882ffee8025" containerID="7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37" exitCode=0 Oct 30 02:36:36.169000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:36.168776 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" Oct 30 02:36:36.169000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:36.168818 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" event={"ID":"3bf1dda1-3b78-4da2-906b-8882ffee8025","Type":"ContainerDied","Data":"7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37"} Oct 30 02:36:36.169000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:36.168881 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf" event={"ID":"3bf1dda1-3b78-4da2-906b-8882ffee8025","Type":"ContainerDied","Data":"babb2cde694de5be9cc17a640702b6b37679149f4bd834abf7f2ee1e132a8e83"} Oct 30 02:36:36.169000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:36.168903 5035 scope.go:122] "RemoveContainer" containerID="7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37" Oct 30 02:36:36.180694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:36.180671 5035 scope.go:122] "RemoveContainer" containerID="7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37" Oct 30 02:36:36.181006 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:36:36.180987 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37\": container with ID starting with 7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37 not found: ID does not exist" containerID="7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37" Oct 30 02:36:36.181090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:36.181015 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37"} err="failed to get container status \"7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37\": rpc error: code = NotFound desc = could not find container \"7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37\": container with ID starting with 7a9973d45e989a0a53bc7b5f7d135a75983400f18ec6b04bf04978b7cf2d4a37 not found: ID does not exist" Oct 30 02:36:36.230172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:36.230138 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf"] Oct 30 02:36:36.241406 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:36.241361 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c75203-8485df74f5-f4jzf"] Oct 30 02:36:37.646433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:37.646396 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf1dda1-3b78-4da2-906b-8882ffee8025" path="/var/lib/kubelet/pods/3bf1dda1-3b78-4da2-906b-8882ffee8025/volumes" Oct 30 02:36:44.295480 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:44.295443 5035 scope.go:122] "RemoveContainer" containerID="1e82963a423512d8876a86feb58a1336a57658ba0efbf098cf49e398a3406659" Oct 30 02:36:47.631887 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:36:47.631845 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:36:48.880579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:48.879371 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw"] Oct 30 02:36:48.912567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:48.892793 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" podUID="dca538fb-fd56-45cb-9d60-dc952c70d5ba" containerName="maas-api" containerID="cri-o://71614811bf2de8a20618bff6161496c974ae8b8050cfe97335200e181296f8d4" gracePeriod=30 Oct 30 02:36:49.366921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:49.366879 5035 generic.go:356] "Generic (PLEG): container finished" podID="dca538fb-fd56-45cb-9d60-dc952c70d5ba" containerID="71614811bf2de8a20618bff6161496c974ae8b8050cfe97335200e181296f8d4" exitCode=0 Oct 30 02:36:49.367588 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:49.367553 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" event={"ID":"dca538fb-fd56-45cb-9d60-dc952c70d5ba","Type":"ContainerDied","Data":"71614811bf2de8a20618bff6161496c974ae8b8050cfe97335200e181296f8d4"} Oct 30 02:36:49.931488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:49.931453 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:36:50.055991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.055877 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-metrics-certs\") pod \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " Oct 30 02:36:50.056170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.056018 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca538fb-fd56-45cb-9d60-dc952c70d5ba-kube-api-access-mfwr6\" (UniqueName: \"kubernetes.io/projected/dca538fb-fd56-45cb-9d60-dc952c70d5ba-kube-api-access-mfwr6\") pod \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " Oct 30 02:36:50.056170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.056086 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca538fb-fd56-45cb-9d60-dc952c70d5ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca538fb-fd56-45cb-9d60-dc952c70d5ba-openshift-service-ca\") pod \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " Oct 30 02:36:50.056311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.056261 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-maas-api-tls\") pod \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\" (UID: \"dca538fb-fd56-45cb-9d60-dc952c70d5ba\") " Oct 30 02:36:50.056496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.056466 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca538fb-fd56-45cb-9d60-dc952c70d5ba-openshift-service-ca" pod "dca538fb-fd56-45cb-9d60-dc952c70d5ba" (UID: "dca538fb-fd56-45cb-9d60-dc952c70d5ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:50.057064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.057033 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca538fb-fd56-45cb-9d60-dc952c70d5ba-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:50.058418 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.058391 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-metrics-certs" pod "dca538fb-fd56-45cb-9d60-dc952c70d5ba" (UID: "dca538fb-fd56-45cb-9d60-dc952c70d5ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:36:50.058677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.058643 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-maas-api-tls" pod "dca538fb-fd56-45cb-9d60-dc952c70d5ba" (UID: "dca538fb-fd56-45cb-9d60-dc952c70d5ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:36:50.058677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.058656 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca538fb-fd56-45cb-9d60-dc952c70d5ba-kube-api-access-mfwr6" pod "dca538fb-fd56-45cb-9d60-dc952c70d5ba" (UID: "dca538fb-fd56-45cb-9d60-dc952c70d5ba"). InnerVolumeSpecName "kube-api-access-mfwr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:50.159288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.159248 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:50.159288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.159283 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfwr6\" (UniqueName: \"kubernetes.io/projected/dca538fb-fd56-45cb-9d60-dc952c70d5ba-kube-api-access-mfwr6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:50.159288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.159293 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca538fb-fd56-45cb-9d60-dc952c70d5ba-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:36:50.382846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.382594 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" event={"ID":"dca538fb-fd56-45cb-9d60-dc952c70d5ba","Type":"ContainerDied","Data":"d4fe478ce26598f150a398e28550d3bdddf8fd2ded539c9af754c4fff4781aa0"} Oct 30 02:36:50.382846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.382629 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" Oct 30 02:36:50.382846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.382793 5035 scope.go:122] "RemoveContainer" containerID="71614811bf2de8a20618bff6161496c974ae8b8050cfe97335200e181296f8d4" Oct 30 02:36:50.467054 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.467013 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw"] Oct 30 02:36:50.484975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:50.484925 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw"] Oct 30 02:36:51.645618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:51.645583 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca538fb-fd56-45cb-9d60-dc952c70d5ba" path="/var/lib/kubelet/pods/dca538fb-fd56-45cb-9d60-dc952c70d5ba/volumes" Oct 30 02:36:52.749439 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:52.749375 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c75203-7d5f897748-r27fw" podUID="dca538fb-fd56-45cb-9d60-dc952c70d5ba" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.235:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:36:55.609178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:55.609134 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2"] Oct 30 02:36:55.609708 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:36:55.609482 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" containerID="cri-o://555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272" gracePeriod=30 Oct 30 02:37:00.922681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.922651 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:37:00.956976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.956937 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-credential-socket\") pod \"d27674de-a2e2-4e4f-82a2-00e44a555270\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " Oct 30 02:37:00.957266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.956995 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-socket\") pod \"d27674de-a2e2-4e4f-82a2-00e44a555270\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " Oct 30 02:37:00.957266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957031 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-podinfo\") pod \"d27674de-a2e2-4e4f-82a2-00e44a555270\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " Oct 30 02:37:00.957266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957077 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27674de-a2e2-4e4f-82a2-00e44a555270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27674de-a2e2-4e4f-82a2-00e44a555270-istiod-ca-cert\") pod \"d27674de-a2e2-4e4f-82a2-00e44a555270\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " Oct 30 02:37:00.957266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957119 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-token\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-token\") pod \"d27674de-a2e2-4e4f-82a2-00e44a555270\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " Oct 30 02:37:00.957266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957159 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-data\") pod \"d27674de-a2e2-4e4f-82a2-00e44a555270\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " Oct 30 02:37:00.957266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957233 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-certs\") pod \"d27674de-a2e2-4e4f-82a2-00e44a555270\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " Oct 30 02:37:00.957664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957308 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-kube-api-access-xvx5n\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-kube-api-access-xvx5n\") pod \"d27674de-a2e2-4e4f-82a2-00e44a555270\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " Oct 30 02:37:00.957664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957338 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-socket" pod "d27674de-a2e2-4e4f-82a2-00e44a555270" (UID: "d27674de-a2e2-4e4f-82a2-00e44a555270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:00.957664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957352 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-envoy\") pod \"d27674de-a2e2-4e4f-82a2-00e44a555270\" (UID: \"d27674de-a2e2-4e4f-82a2-00e44a555270\") " Oct 30 02:37:00.957664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957474 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-credential-socket" pod "d27674de-a2e2-4e4f-82a2-00e44a555270" (UID: "d27674de-a2e2-4e4f-82a2-00e44a555270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:00.957993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957955 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-certs" pod "d27674de-a2e2-4e4f-82a2-00e44a555270" (UID: "d27674de-a2e2-4e4f-82a2-00e44a555270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:00.957993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.957970 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27674de-a2e2-4e4f-82a2-00e44a555270-istiod-ca-cert" pod "d27674de-a2e2-4e4f-82a2-00e44a555270" (UID: "d27674de-a2e2-4e4f-82a2-00e44a555270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:37:00.958134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.958102 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-data" pod "d27674de-a2e2-4e4f-82a2-00e44a555270" (UID: "d27674de-a2e2-4e4f-82a2-00e44a555270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:00.958370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.958348 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:37:00.958445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.958377 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:37:00.958445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.958394 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27674de-a2e2-4e4f-82a2-00e44a555270-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:37:00.958445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.958408 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:37:00.958445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.958422 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:37:00.960048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.960015 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-token" pod "d27674de-a2e2-4e4f-82a2-00e44a555270" (UID: "d27674de-a2e2-4e4f-82a2-00e44a555270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:37:00.960170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.960073 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-envoy" pod "d27674de-a2e2-4e4f-82a2-00e44a555270" (UID: "d27674de-a2e2-4e4f-82a2-00e44a555270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:00.960170 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.960088 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-podinfo" pod "d27674de-a2e2-4e4f-82a2-00e44a555270" (UID: "d27674de-a2e2-4e4f-82a2-00e44a555270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:37:00.960257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:00.960183 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-kube-api-access-xvx5n" pod "d27674de-a2e2-4e4f-82a2-00e44a555270" (UID: "d27674de-a2e2-4e4f-82a2-00e44a555270"). InnerVolumeSpecName "kube-api-access-xvx5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:37:01.060342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.060303 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvx5n\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-kube-api-access-xvx5n\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:37:01.060342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.060340 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:37:01.060342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.060350 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:37:01.060649 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.060359 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27674de-a2e2-4e4f-82a2-00e44a555270-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:37:01.486000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.485960 5035 generic.go:356] "Generic (PLEG): container finished" podID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerID="555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272" exitCode=0 Oct 30 02:37:01.486198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.486022 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" Oct 30 02:37:01.486198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.486038 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" event={"ID":"d27674de-a2e2-4e4f-82a2-00e44a555270","Type":"ContainerDied","Data":"555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272"} Oct 30 02:37:01.486198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.486101 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2" event={"ID":"d27674de-a2e2-4e4f-82a2-00e44a555270","Type":"ContainerDied","Data":"0fe61e5677ff98c1ab2018db1f5025e068a6b2844671ffce6b5fd61f7413dd1a"} Oct 30 02:37:01.486198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.486125 5035 scope.go:122] "RemoveContainer" containerID="555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272" Oct 30 02:37:01.498822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.498794 5035 scope.go:122] "RemoveContainer" containerID="e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3" Oct 30 02:37:01.510143 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.510118 5035 scope.go:122] "RemoveContainer" containerID="555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272" Oct 30 02:37:01.510457 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:37:01.510426 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272\": container with ID starting with 555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272 not found: ID does not exist" containerID="555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272" Oct 30 02:37:01.510513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.510468 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272"} err="failed to get container status \"555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272\": rpc error: code = NotFound desc = could not find container \"555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272\": container with ID starting with 555444cdd9f17ce11d4570599f12cb224d5efc4a00be4f5fbed254eefa603272 not found: ID does not exist" Oct 30 02:37:01.510513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.510487 5035 scope.go:122] "RemoveContainer" containerID="e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3" Oct 30 02:37:01.510729 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:37:01.510712 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3\": container with ID starting with e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3 not found: ID does not exist" containerID="e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3" Oct 30 02:37:01.510774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.510738 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3"} err="failed to get container status \"e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3\": rpc error: code = NotFound desc = could not find container \"e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3\": container with ID starting with e8b2af47eb62b8852c04204c554fe0a68fb0043d69a5743fe335995f60bcceb3 not found: ID does not exist" Oct 30 02:37:01.552544 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.552500 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2"] Oct 30 02:37:01.564497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.564460 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c75203-openshift-default-5db7f76f89-8mxc2"] Oct 30 02:37:01.645733 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:37:01.645700 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" path="/var/lib/kubelet/pods/d27674de-a2e2-4e4f-82a2-00e44a555270/volumes" Oct 30 02:37:17.663287 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:37:17.663241 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:37:47.630793 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:37:47.630749 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:38:17.571349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:17.571301 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t"] Oct 30 02:38:17.637961 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:38:17.637920 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:38:22.572652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:22.572601 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" podUID="0b3d0792-204a-4487-9ea1-efb6b6616c70" containerName="payload-pre-processing" containerID="cri-o://f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602" gracePeriod=25 Oct 30 02:38:24.145434 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.145396 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt"] Oct 30 02:38:24.145906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.145692 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" podUID="e98d14cd-45be-457a-a181-3a96931d419e" containerName="maas-api" containerID="cri-o://5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb" gracePeriod=30 Oct 30 02:38:24.744065 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.744030 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:38:24.840012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.839947 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-maas-api-tls\") pod \"e98d14cd-45be-457a-a181-3a96931d419e\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " Oct 30 02:38:24.840229 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.840091 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98d14cd-45be-457a-a181-3a96931d419e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98d14cd-45be-457a-a181-3a96931d419e-openshift-service-ca\") pod \"e98d14cd-45be-457a-a181-3a96931d419e\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " Oct 30 02:38:24.840229 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.840132 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98d14cd-45be-457a-a181-3a96931d419e-kube-api-access-b2pfm\" (UniqueName: \"kubernetes.io/projected/e98d14cd-45be-457a-a181-3a96931d419e-kube-api-access-b2pfm\") pod \"e98d14cd-45be-457a-a181-3a96931d419e\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " Oct 30 02:38:24.840361 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.840230 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-metrics-certs\") pod \"e98d14cd-45be-457a-a181-3a96931d419e\" (UID: \"e98d14cd-45be-457a-a181-3a96931d419e\") " Oct 30 02:38:24.840640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.840606 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98d14cd-45be-457a-a181-3a96931d419e-openshift-service-ca" pod "e98d14cd-45be-457a-a181-3a96931d419e" (UID: "e98d14cd-45be-457a-a181-3a96931d419e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:38:24.841043 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.841013 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98d14cd-45be-457a-a181-3a96931d419e-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:38:24.842715 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.842685 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-maas-api-tls" pod "e98d14cd-45be-457a-a181-3a96931d419e" (UID: "e98d14cd-45be-457a-a181-3a96931d419e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:38:24.843049 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.843024 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-metrics-certs" pod "e98d14cd-45be-457a-a181-3a96931d419e" (UID: "e98d14cd-45be-457a-a181-3a96931d419e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:38:24.843131 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.843037 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98d14cd-45be-457a-a181-3a96931d419e-kube-api-access-b2pfm" pod "e98d14cd-45be-457a-a181-3a96931d419e" (UID: "e98d14cd-45be-457a-a181-3a96931d419e"). InnerVolumeSpecName "kube-api-access-b2pfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:24.942288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.942244 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:38:24.942288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.942280 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2pfm\" (UniqueName: \"kubernetes.io/projected/e98d14cd-45be-457a-a181-3a96931d419e-kube-api-access-b2pfm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:38:24.942288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:24.942290 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98d14cd-45be-457a-a181-3a96931d419e-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:38:25.262443 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.262400 5035 generic.go:356] "Generic (PLEG): container finished" podID="e98d14cd-45be-457a-a181-3a96931d419e" containerID="5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb" exitCode=0 Oct 30 02:38:25.263163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.262465 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" Oct 30 02:38:25.263163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.262743 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" event={"ID":"e98d14cd-45be-457a-a181-3a96931d419e","Type":"ContainerDied","Data":"5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb"} Oct 30 02:38:25.263163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.262798 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt" event={"ID":"e98d14cd-45be-457a-a181-3a96931d419e","Type":"ContainerDied","Data":"bb1c3a50b5f25c73b99061968762d7117bcb6baf2196e98e1e7f22a535056f2f"} Oct 30 02:38:25.263163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.262831 5035 scope.go:122] "RemoveContainer" containerID="5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb" Oct 30 02:38:25.401764 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.401730 5035 scope.go:122] "RemoveContainer" containerID="5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb" Oct 30 02:38:25.402459 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:38:25.402424 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb\": container with ID starting with 5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb not found: ID does not exist" containerID="5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb" Oct 30 02:38:25.402766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.402459 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb"} err="failed to get container status \"5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb\": rpc error: code = NotFound desc = could not find container \"5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb\": container with ID starting with 5cbb9e94eecfe0bab0489d4fb3eeb098b6fa798e340304d41f913ae1f9d19fcb not found: ID does not exist" Oct 30 02:38:25.403846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.403814 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt"] Oct 30 02:38:25.412139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.412105 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-52d131-7d467cbfbc-9dpmt"] Oct 30 02:38:25.645647 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:25.645567 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98d14cd-45be-457a-a181-3a96931d419e" path="/var/lib/kubelet/pods/e98d14cd-45be-457a-a181-3a96931d419e/volumes" Oct 30 02:38:26.011712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:26.011675 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" podUID="0b3d0792-204a-4487-9ea1-efb6b6616c70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:38:36.012160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:36.012106 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" podUID="0b3d0792-204a-4487-9ea1-efb6b6616c70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:38:42.865226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:42.865137 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:38:43.024627 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.024577 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b3d0792-204a-4487-9ea1-efb6b6616c70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3d0792-204a-4487-9ea1-efb6b6616c70-plugins-config-volume\") pod \"0b3d0792-204a-4487-9ea1-efb6b6616c70\" (UID: \"0b3d0792-204a-4487-9ea1-efb6b6616c70\") " Oct 30 02:38:43.024817 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.024663 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3d0792-204a-4487-9ea1-efb6b6616c70-kube-api-access-jrrlm\" (UniqueName: \"kubernetes.io/projected/0b3d0792-204a-4487-9ea1-efb6b6616c70-kube-api-access-jrrlm\") pod \"0b3d0792-204a-4487-9ea1-efb6b6616c70\" (UID: \"0b3d0792-204a-4487-9ea1-efb6b6616c70\") " Oct 30 02:38:43.025057 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.025029 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3d0792-204a-4487-9ea1-efb6b6616c70-plugins-config-volume" pod "0b3d0792-204a-4487-9ea1-efb6b6616c70" (UID: "0b3d0792-204a-4487-9ea1-efb6b6616c70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:38:43.025262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.025241 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b3d0792-204a-4487-9ea1-efb6b6616c70-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:38:43.027211 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.027182 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3d0792-204a-4487-9ea1-efb6b6616c70-kube-api-access-jrrlm" pod "0b3d0792-204a-4487-9ea1-efb6b6616c70" (UID: "0b3d0792-204a-4487-9ea1-efb6b6616c70"). InnerVolumeSpecName "kube-api-access-jrrlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:43.127247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.127150 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrrlm\" (UniqueName: \"kubernetes.io/projected/0b3d0792-204a-4487-9ea1-efb6b6616c70-kube-api-access-jrrlm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:38:43.501017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.500973 5035 generic.go:356] "Generic (PLEG): container finished" podID="0b3d0792-204a-4487-9ea1-efb6b6616c70" containerID="f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602" exitCode=0 Oct 30 02:38:43.501243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.501023 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" Oct 30 02:38:43.501243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.501055 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" event={"ID":"0b3d0792-204a-4487-9ea1-efb6b6616c70","Type":"ContainerDied","Data":"f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602"} Oct 30 02:38:43.501243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.501145 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t" event={"ID":"0b3d0792-204a-4487-9ea1-efb6b6616c70","Type":"ContainerDied","Data":"0304bde404351a224b08893365e1b09e2c849a9c02c930c839aa60b46149a7ad"} Oct 30 02:38:43.501243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.501170 5035 scope.go:122] "RemoveContainer" containerID="f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602" Oct 30 02:38:43.513076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.513053 5035 scope.go:122] "RemoveContainer" containerID="f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602" Oct 30 02:38:43.513450 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:38:43.513423 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602\": container with ID starting with f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602 not found: ID does not exist" containerID="f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602" Oct 30 02:38:43.513585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.513455 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602"} err="failed to get container status \"f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602\": rpc error: code = NotFound desc = could not find container \"f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602\": container with ID starting with f4d113af100d55be8dea14a0c252268003f42c1347b8e00ae63d5947690e4602 not found: ID does not exist" Oct 30 02:38:43.553893 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.553850 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t"] Oct 30 02:38:43.565224 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.565183 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52d131-58d9cf5ccf-ngs9t"] Oct 30 02:38:43.644815 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:43.644777 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b3d0792-204a-4487-9ea1-efb6b6616c70" path="/var/lib/kubelet/pods/0b3d0792-204a-4487-9ea1-efb6b6616c70/volumes" Oct 30 02:38:44.063654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:44.063623 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:38:44.074449 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:44.074421 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:38:44.079233 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:44.079205 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:38:44.129894 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:44.129867 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:38:44.141070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:44.141043 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:38:44.145834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:44.145807 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:38:47.632403 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:38:47.632357 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:38:55.864186 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:55.864145 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh"] Oct 30 02:38:55.864810 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:38:55.864394 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" containerID="cri-o://c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c" gracePeriod=30 Oct 30 02:39:00.368452 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:00.368415 5035 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:39:00.368923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:00.368474 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:39:01.169422 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.169345 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:39:01.265078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265041 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-podinfo\") pod \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " Oct 30 02:39:01.265308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265128 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-kube-api-access-cfg74\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-kube-api-access-cfg74\") pod \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " Oct 30 02:39:01.265308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265156 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-socket\") pod \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " Oct 30 02:39:01.265308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265180 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-credential-socket\") pod \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " Oct 30 02:39:01.265308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265230 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-envoy\") pod \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " Oct 30 02:39:01.265308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265280 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-token\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-token\") pod \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " Oct 30 02:39:01.265648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265320 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istiod-ca-cert\") pod \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " Oct 30 02:39:01.265648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265471 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-data\") pod \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " Oct 30 02:39:01.265648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265575 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-certs\") pod \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\" (UID: \"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc\") " Oct 30 02:39:01.265648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265523 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-socket" pod "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" (UID: "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:01.265964 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.265937 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istiod-ca-cert" pod "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" (UID: "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:01.266094 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.266065 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-data" pod "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" (UID: "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:01.266294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.266273 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-certs" pod "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" (UID: "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:01.266373 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.266305 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-credential-socket" pod "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" (UID: "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:01.266451 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.266430 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:01.266649 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.266630 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:01.266765 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.266751 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:01.266862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.266851 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:01.267988 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.267958 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-podinfo" pod "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" (UID: "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:39:01.268346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.268320 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-kube-api-access-cfg74" pod "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" (UID: "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc"). InnerVolumeSpecName "kube-api-access-cfg74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:01.268425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.268373 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-token" pod "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" (UID: "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:01.268718 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.268697 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-envoy" pod "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" (UID: "8c1c0f8a-ab2a-43dc-bda9-e8461b76facc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:01.368831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.368788 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfg74\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-kube-api-access-cfg74\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:01.368831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.368827 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:01.368831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.368837 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:01.369333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.368846 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:01.369333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.368856 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:01.686129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.686092 5035 generic.go:356] "Generic (PLEG): container finished" podID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerID="c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c" exitCode=0 Oct 30 02:39:01.686330 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.686154 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" Oct 30 02:39:01.686330 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.686178 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" event={"ID":"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc","Type":"ContainerDied","Data":"c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c"} Oct 30 02:39:01.686330 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.686239 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh" event={"ID":"8c1c0f8a-ab2a-43dc-bda9-e8461b76facc","Type":"ContainerDied","Data":"f3910eeed8f57266fd973f8e2afaf6e5c73aab69d062c97a01d0ee280c835414"} Oct 30 02:39:01.686330 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.686261 5035 scope.go:122] "RemoveContainer" containerID="c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c" Oct 30 02:39:01.697951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.697923 5035 scope.go:122] "RemoveContainer" containerID="c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c" Oct 30 02:39:01.698328 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:39:01.698290 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c\": container with ID starting with c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c not found: ID does not exist" containerID="c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c" Oct 30 02:39:01.698445 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.698337 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c"} err="failed to get container status \"c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c\": rpc error: code = NotFound desc = could not find container \"c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c\": container with ID starting with c21b8005c709f1700c72d922cad20ac098eb3569331622faedb340b6e1888b6c not found: ID does not exist" Oct 30 02:39:01.741453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.741338 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh"] Oct 30 02:39:01.747265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:01.747232 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-2s6zh"] Oct 30 02:39:03.645477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:03.645441 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" path="/var/lib/kubelet/pods/8c1c0f8a-ab2a-43dc-bda9-e8461b76facc/volumes" Oct 30 02:39:17.630227 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:39:17.630181 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:39:43.274550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:43.264855 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58"] Oct 30 02:39:43.277724 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:43.277679 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" containerID="cri-o://987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6" gracePeriod=30 Oct 30 02:39:44.652308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:44.652272 5035 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:39:44.652822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:44.652362 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:39:47.632463 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:39:47.632417 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:39:48.587215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.587182 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:39:48.660294 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.660247 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-envoy\") pod \"a4b60bc0-359c-4074-a4ff-7f561627df13\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " Oct 30 02:39:48.660831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.660304 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-data\") pod \"a4b60bc0-359c-4074-a4ff-7f561627df13\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " Oct 30 02:39:48.660831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.660350 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-podinfo\") pod \"a4b60bc0-359c-4074-a4ff-7f561627df13\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " Oct 30 02:39:48.660831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.660391 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-kube-api-access-mzwxx\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-kube-api-access-mzwxx\") pod \"a4b60bc0-359c-4074-a4ff-7f561627df13\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " Oct 30 02:39:48.660831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.660605 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-credential-socket\") pod \"a4b60bc0-359c-4074-a4ff-7f561627df13\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " Oct 30 02:39:48.660831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.660637 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-token\") pod \"a4b60bc0-359c-4074-a4ff-7f561627df13\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " Oct 30 02:39:48.660831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.660703 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-certs\") pod \"a4b60bc0-359c-4074-a4ff-7f561627df13\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " Oct 30 02:39:48.660831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.660787 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b60bc0-359c-4074-a4ff-7f561627df13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60bc0-359c-4074-a4ff-7f561627df13-istiod-ca-cert\") pod \"a4b60bc0-359c-4074-a4ff-7f561627df13\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " Oct 30 02:39:48.661313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.660852 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-socket\") pod \"a4b60bc0-359c-4074-a4ff-7f561627df13\" (UID: \"a4b60bc0-359c-4074-a4ff-7f561627df13\") " Oct 30 02:39:48.661390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.661357 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-data" pod "a4b60bc0-359c-4074-a4ff-7f561627df13" (UID: "a4b60bc0-359c-4074-a4ff-7f561627df13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:48.661565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.661500 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-credential-socket" pod "a4b60bc0-359c-4074-a4ff-7f561627df13" (UID: "a4b60bc0-359c-4074-a4ff-7f561627df13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:48.661862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.661840 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-certs" pod "a4b60bc0-359c-4074-a4ff-7f561627df13" (UID: "a4b60bc0-359c-4074-a4ff-7f561627df13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:48.661975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.661905 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:48.661975 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.661927 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:48.662109 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.662065 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-socket" pod "a4b60bc0-359c-4074-a4ff-7f561627df13" (UID: "a4b60bc0-359c-4074-a4ff-7f561627df13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:48.662183 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.662162 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b60bc0-359c-4074-a4ff-7f561627df13-istiod-ca-cert" pod "a4b60bc0-359c-4074-a4ff-7f561627df13" (UID: "a4b60bc0-359c-4074-a4ff-7f561627df13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:48.663978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.663947 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-token" pod "a4b60bc0-359c-4074-a4ff-7f561627df13" (UID: "a4b60bc0-359c-4074-a4ff-7f561627df13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:48.664825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.664779 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-envoy" pod "a4b60bc0-359c-4074-a4ff-7f561627df13" (UID: "a4b60bc0-359c-4074-a4ff-7f561627df13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:48.673137 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.673099 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-kube-api-access-mzwxx" pod "a4b60bc0-359c-4074-a4ff-7f561627df13" (UID: "a4b60bc0-359c-4074-a4ff-7f561627df13"). InnerVolumeSpecName "kube-api-access-mzwxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:48.673299 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.673212 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-podinfo" pod "a4b60bc0-359c-4074-a4ff-7f561627df13" (UID: "a4b60bc0-359c-4074-a4ff-7f561627df13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:39:48.763357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.763311 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:48.763357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.763351 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:48.763357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.763361 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzwxx\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-kube-api-access-mzwxx\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:48.763676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.763372 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b60bc0-359c-4074-a4ff-7f561627df13-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:48.763676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.763380 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:48.763676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.763390 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b60bc0-359c-4074-a4ff-7f561627df13-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:48.763676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:48.763399 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b60bc0-359c-4074-a4ff-7f561627df13-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:39:49.161348 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.161307 5035 generic.go:356] "Generic (PLEG): container finished" podID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerID="987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6" exitCode=0 Oct 30 02:39:49.161567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.161358 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" Oct 30 02:39:49.161567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.161395 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" event={"ID":"a4b60bc0-359c-4074-a4ff-7f561627df13","Type":"ContainerDied","Data":"987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6"} Oct 30 02:39:49.161567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.161454 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58" event={"ID":"a4b60bc0-359c-4074-a4ff-7f561627df13","Type":"ContainerDied","Data":"fa77238dbcdbdb705fd4f89fc974d63d6e3ebe9786520eeb0cfc19cac32680b3"} Oct 30 02:39:49.161567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.161477 5035 scope.go:122] "RemoveContainer" containerID="987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6" Oct 30 02:39:49.175176 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.175124 5035 scope.go:122] "RemoveContainer" containerID="987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6" Oct 30 02:39:49.176501 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:39:49.176425 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6\": container with ID starting with 987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6 not found: ID does not exist" containerID="987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6" Oct 30 02:39:49.177038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.176554 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6"} err="failed to get container status \"987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6\": rpc error: code = NotFound desc = could not find container \"987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6\": container with ID starting with 987af65743f8bd0470196cd941568483aace7f5e7e8fa00a4521383258edf5b6 not found: ID does not exist" Oct 30 02:39:49.209625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.209585 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv"] Oct 30 02:39:49.216877 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.216845 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58"] Oct 30 02:39:49.221890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.221858 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-6zz58"] Oct 30 02:39:49.645479 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:49.645439 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" path="/var/lib/kubelet/pods/a4b60bc0-359c-4074-a4ff-7f561627df13/volumes" Oct 30 02:39:54.210011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:39:54.209908 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" podUID="6c0e8297-c571-4105-b69a-c7eb9c3938be" containerName="payload-processing" containerID="cri-o://d1af9cc72a9a0255020e279ffaa2cdd2dbabf4835950a9d676e310431ce30975" gracePeriod=25 Oct 30 02:40:02.279613 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:02.279571 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" podUID="6c0e8297-c571-4105-b69a-c7eb9c3938be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:02.692830 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:02.692735 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d"] Oct 30 02:40:02.693012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:02.692959 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" containerID="cri-o://6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94" gracePeriod=30 Oct 30 02:40:04.439017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:04.438975 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r"] Oct 30 02:40:04.439573 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:04.439231 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" containerID="cri-o://ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab" gracePeriod=30 Oct 30 02:40:06.428863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:06.428819 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r"] Oct 30 02:40:06.462432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:06.462387 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn"] Oct 30 02:40:06.567936 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:06.567897 5035 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:40:06.568147 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:06.567973 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:40:08.010614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.010487 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:40:08.116438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.116387 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-data\") pod \"aa390600-ff7d-4f05-bc0c-9ada10153b57\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " Oct 30 02:40:08.116678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.116468 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa390600-ff7d-4f05-bc0c-9ada10153b57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa390600-ff7d-4f05-bc0c-9ada10153b57-istiod-ca-cert\") pod \"aa390600-ff7d-4f05-bc0c-9ada10153b57\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " Oct 30 02:40:08.116678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.116563 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-socket\") pod \"aa390600-ff7d-4f05-bc0c-9ada10153b57\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " Oct 30 02:40:08.116678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.116624 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-certs\") pod \"aa390600-ff7d-4f05-bc0c-9ada10153b57\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " Oct 30 02:40:08.116822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.116726 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-token\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-token\") pod \"aa390600-ff7d-4f05-bc0c-9ada10153b57\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " Oct 30 02:40:08.116822 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.116763 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-data" pod "aa390600-ff7d-4f05-bc0c-9ada10153b57" (UID: "aa390600-ff7d-4f05-bc0c-9ada10153b57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:08.116930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.116819 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-envoy\") pod \"aa390600-ff7d-4f05-bc0c-9ada10153b57\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " Oct 30 02:40:08.116930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.116903 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-credential-socket\") pod \"aa390600-ff7d-4f05-bc0c-9ada10153b57\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " Oct 30 02:40:08.117061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.117002 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-socket" pod "aa390600-ff7d-4f05-bc0c-9ada10153b57" (UID: "aa390600-ff7d-4f05-bc0c-9ada10153b57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:08.117141 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.117066 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-certs" pod "aa390600-ff7d-4f05-bc0c-9ada10153b57" (UID: "aa390600-ff7d-4f05-bc0c-9ada10153b57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:08.117325 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.117301 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa390600-ff7d-4f05-bc0c-9ada10153b57-istiod-ca-cert" pod "aa390600-ff7d-4f05-bc0c-9ada10153b57" (UID: "aa390600-ff7d-4f05-bc0c-9ada10153b57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:08.117325 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.117312 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-credential-socket" pod "aa390600-ff7d-4f05-bc0c-9ada10153b57" (UID: "aa390600-ff7d-4f05-bc0c-9ada10153b57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:08.117567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.117484 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-podinfo\") pod \"aa390600-ff7d-4f05-bc0c-9ada10153b57\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " Oct 30 02:40:08.117824 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.117799 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-kube-api-access-rdvkk\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-kube-api-access-rdvkk\") pod \"aa390600-ff7d-4f05-bc0c-9ada10153b57\" (UID: \"aa390600-ff7d-4f05-bc0c-9ada10153b57\") " Oct 30 02:40:08.118565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.118542 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:08.118705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.118572 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa390600-ff7d-4f05-bc0c-9ada10153b57-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:08.118705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.118585 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:08.118705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.118601 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:08.118705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.118615 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:08.119821 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.119793 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-token" pod "aa390600-ff7d-4f05-bc0c-9ada10153b57" (UID: "aa390600-ff7d-4f05-bc0c-9ada10153b57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:08.119980 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.119953 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-envoy" pod "aa390600-ff7d-4f05-bc0c-9ada10153b57" (UID: "aa390600-ff7d-4f05-bc0c-9ada10153b57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:08.120144 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.120117 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-podinfo" pod "aa390600-ff7d-4f05-bc0c-9ada10153b57" (UID: "aa390600-ff7d-4f05-bc0c-9ada10153b57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:40:08.120961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.120928 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-kube-api-access-rdvkk" pod "aa390600-ff7d-4f05-bc0c-9ada10153b57" (UID: "aa390600-ff7d-4f05-bc0c-9ada10153b57"). InnerVolumeSpecName "kube-api-access-rdvkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:08.190882 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.190838 5035 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:40:08.191069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.190910 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:40:08.220331 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.220294 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:08.220331 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.220327 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:08.220331 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.220337 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa390600-ff7d-4f05-bc0c-9ada10153b57-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:08.220694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.220346 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvkk\" (UniqueName: \"kubernetes.io/projected/aa390600-ff7d-4f05-bc0c-9ada10153b57-kube-api-access-rdvkk\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:08.340312 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.340204 5035 generic.go:356] "Generic (PLEG): container finished" podID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerID="6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94" exitCode=0 Oct 30 02:40:08.340312 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.340254 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" Oct 30 02:40:08.340312 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.340291 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" event={"ID":"aa390600-ff7d-4f05-bc0c-9ada10153b57","Type":"ContainerDied","Data":"6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94"} Oct 30 02:40:08.340665 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.340356 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d" event={"ID":"aa390600-ff7d-4f05-bc0c-9ada10153b57","Type":"ContainerDied","Data":"b8e3dda40dc7e21f4bc14af8b687ca5fe7d80505f1c7f4b5faee2c704cf2b12b"} Oct 30 02:40:08.340665 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.340386 5035 scope.go:122] "RemoveContainer" containerID="6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94" Oct 30 02:40:08.353467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.353440 5035 scope.go:122] "RemoveContainer" containerID="6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94" Oct 30 02:40:08.353837 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:40:08.353809 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94\": container with ID starting with 6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94 not found: ID does not exist" containerID="6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94" Oct 30 02:40:08.353938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.353846 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94"} err="failed to get container status \"6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94\": rpc error: code = NotFound desc = could not find container \"6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94\": container with ID starting with 6a4c930208691be2bd05c2b4da06301f87196389db46ff7e8c499f0e6c591e94 not found: ID does not exist" Oct 30 02:40:08.394353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.394315 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d"] Oct 30 02:40:08.399737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:08.399704 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-rjn2d"] Oct 30 02:40:09.648243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.648205 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" path="/var/lib/kubelet/pods/aa390600-ff7d-4f05-bc0c-9ada10153b57/volumes" Oct 30 02:40:09.728172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.728141 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 02:40:09.843900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.843859 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-envoy\") pod \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " Oct 30 02:40:09.844111 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.843919 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istiod-ca-cert\") pod \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " Oct 30 02:40:09.844111 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.843963 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-credential-socket\") pod \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " Oct 30 02:40:09.844111 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.843995 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-certs\") pod \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " Oct 30 02:40:09.844111 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844015 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-data\") pod \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " Oct 30 02:40:09.844344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844120 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-socket\") pod \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " Oct 30 02:40:09.844344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844288 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-podinfo\") pod \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " Oct 30 02:40:09.844344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844327 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-token\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-token\") pod \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " Oct 30 02:40:09.844496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844334 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-data" pod "b5a7ded3-59ff-4e1c-bb3a-00956659d646" (UID: "b5a7ded3-59ff-4e1c-bb3a-00956659d646"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:09.844496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844351 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-credential-socket" pod "b5a7ded3-59ff-4e1c-bb3a-00956659d646" (UID: "b5a7ded3-59ff-4e1c-bb3a-00956659d646"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:09.844496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844388 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-certs" pod "b5a7ded3-59ff-4e1c-bb3a-00956659d646" (UID: "b5a7ded3-59ff-4e1c-bb3a-00956659d646"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:09.844496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844430 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-socket" pod "b5a7ded3-59ff-4e1c-bb3a-00956659d646" (UID: "b5a7ded3-59ff-4e1c-bb3a-00956659d646"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:09.844496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844466 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-kube-api-access-9jh9g\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-kube-api-access-9jh9g\") pod \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\" (UID: \"b5a7ded3-59ff-4e1c-bb3a-00956659d646\") " Oct 30 02:40:09.844773 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.844740 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istiod-ca-cert" pod "b5a7ded3-59ff-4e1c-bb3a-00956659d646" (UID: "b5a7ded3-59ff-4e1c-bb3a-00956659d646"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:09.845168 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.845139 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:09.845316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.845172 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:09.845316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.845188 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:09.845316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.845203 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:09.845316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.845218 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:09.846488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.846461 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-envoy" pod "b5a7ded3-59ff-4e1c-bb3a-00956659d646" (UID: "b5a7ded3-59ff-4e1c-bb3a-00956659d646"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:09.846488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.846478 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-podinfo" pod "b5a7ded3-59ff-4e1c-bb3a-00956659d646" (UID: "b5a7ded3-59ff-4e1c-bb3a-00956659d646"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:40:09.846660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.846522 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-kube-api-access-9jh9g" pod "b5a7ded3-59ff-4e1c-bb3a-00956659d646" (UID: "b5a7ded3-59ff-4e1c-bb3a-00956659d646"). InnerVolumeSpecName "kube-api-access-9jh9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:09.846712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.846693 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-token" pod "b5a7ded3-59ff-4e1c-bb3a-00956659d646" (UID: "b5a7ded3-59ff-4e1c-bb3a-00956659d646"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:09.946696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.946656 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:09.946696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.946687 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:09.946696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.946698 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jh9g\" (UniqueName: \"kubernetes.io/projected/b5a7ded3-59ff-4e1c-bb3a-00956659d646-kube-api-access-9jh9g\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:09.947026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:09.946710 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a7ded3-59ff-4e1c-bb3a-00956659d646-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:10.362672 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.362628 5035 generic.go:356] "Generic (PLEG): container finished" podID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerID="ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab" exitCode=0 Oct 30 02:40:10.362855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.362679 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" Oct 30 02:40:10.362855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.362704 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" event={"ID":"b5a7ded3-59ff-4e1c-bb3a-00956659d646","Type":"ContainerDied","Data":"ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab"} Oct 30 02:40:10.362855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.362766 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r" event={"ID":"b5a7ded3-59ff-4e1c-bb3a-00956659d646","Type":"ContainerDied","Data":"06d352169b10e92993ac9db3c530bc4f37429f2a8d5e94ae34743fa9a6f40f57"} Oct 30 02:40:10.362855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.362792 5035 scope.go:122] "RemoveContainer" containerID="ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab" Oct 30 02:40:10.375454 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.375434 5035 scope.go:122] "RemoveContainer" containerID="f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded" Oct 30 02:40:10.386203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.386178 5035 scope.go:122] "RemoveContainer" containerID="ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab" Oct 30 02:40:10.386538 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:40:10.386503 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab\": container with ID starting with ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab not found: ID does not exist" containerID="ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab" Oct 30 02:40:10.386607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.386550 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab"} err="failed to get container status \"ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab\": rpc error: code = NotFound desc = could not find container \"ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab\": container with ID starting with ec63a657b5fe936d6edf2cbb217962fa4544df6e29435f88ac01b32de8f618ab not found: ID does not exist" Oct 30 02:40:10.386607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.386569 5035 scope.go:122] "RemoveContainer" containerID="f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded" Oct 30 02:40:10.386850 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:40:10.386831 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded\": container with ID starting with f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded not found: ID does not exist" containerID="f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded" Oct 30 02:40:10.386893 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.386856 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded"} err="failed to get container status \"f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded\": rpc error: code = NotFound desc = could not find container \"f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded\": container with ID starting with f9e53eca6c722a57bedf8f8a6104455e4a50c4c81bc35487cd5da463d5929ded not found: ID does not exist" Oct 30 02:40:10.430129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.430096 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r"] Oct 30 02:40:10.434102 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:10.434071 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-w424r"] Oct 30 02:40:11.429933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:11.429892 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" podUID="b6267677-8af0-44d5-9d95-8e8eb629463e" containerName="payload-processing" containerID="cri-o://5bc0bad9a453c4a2ac671e105a4c8073503ff578b66ebec6a26086c49648e3c6" gracePeriod=25 Oct 30 02:40:11.463660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:11.463611 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" podUID="6c9850ac-3004-42da-8ea5-d005e5f5bbca" containerName="payload-pre-processing" containerID="cri-o://6d41cf34e8f4c2974bb18ffe21197f8d96bc27ecc4104e440f26f4edee183c26" gracePeriod=25 Oct 30 02:40:11.645603 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:11.645566 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" path="/var/lib/kubelet/pods/b5a7ded3-59ff-4e1c-bb3a-00956659d646/volumes" Oct 30 02:40:12.279642 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:12.279603 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" podUID="6c0e8297-c571-4105-b69a-c7eb9c3938be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:14.410457 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:14.410415 5035 generic.go:356] "Generic (PLEG): container finished" podID="6c0e8297-c571-4105-b69a-c7eb9c3938be" containerID="d1af9cc72a9a0255020e279ffaa2cdd2dbabf4835950a9d676e310431ce30975" exitCode=0 Oct 30 02:40:14.410906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:14.410503 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" event={"ID":"6c0e8297-c571-4105-b69a-c7eb9c3938be","Type":"ContainerDied","Data":"d1af9cc72a9a0255020e279ffaa2cdd2dbabf4835950a9d676e310431ce30975"} Oct 30 02:40:14.514147 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:14.514118 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:40:14.612677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:14.612625 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0e8297-c571-4105-b69a-c7eb9c3938be-kube-api-access-284l8\" (UniqueName: \"kubernetes.io/projected/6c0e8297-c571-4105-b69a-c7eb9c3938be-kube-api-access-284l8\") pod \"6c0e8297-c571-4105-b69a-c7eb9c3938be\" (UID: \"6c0e8297-c571-4105-b69a-c7eb9c3938be\") " Oct 30 02:40:14.612912 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:14.612805 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0e8297-c571-4105-b69a-c7eb9c3938be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0e8297-c571-4105-b69a-c7eb9c3938be-plugins-config-volume\") pod \"6c0e8297-c571-4105-b69a-c7eb9c3938be\" (UID: \"6c0e8297-c571-4105-b69a-c7eb9c3938be\") " Oct 30 02:40:14.613185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:14.613160 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0e8297-c571-4105-b69a-c7eb9c3938be-plugins-config-volume" pod "6c0e8297-c571-4105-b69a-c7eb9c3938be" (UID: "6c0e8297-c571-4105-b69a-c7eb9c3938be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:14.613360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:14.613345 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0e8297-c571-4105-b69a-c7eb9c3938be-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:14.615054 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:14.615029 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0e8297-c571-4105-b69a-c7eb9c3938be-kube-api-access-284l8" pod "6c0e8297-c571-4105-b69a-c7eb9c3938be" (UID: "6c0e8297-c571-4105-b69a-c7eb9c3938be"). InnerVolumeSpecName "kube-api-access-284l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:14.715818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:14.715603 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284l8\" (UniqueName: \"kubernetes.io/projected/6c0e8297-c571-4105-b69a-c7eb9c3938be-kube-api-access-284l8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:15.420187 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:15.420152 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" Oct 30 02:40:15.420656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:15.420148 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv" event={"ID":"6c0e8297-c571-4105-b69a-c7eb9c3938be","Type":"ContainerDied","Data":"4de9588002fcc3bb56d2ee1021ace3fc4b0ca72897eda40b2cfb580a468e876b"} Oct 30 02:40:15.420656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:15.420320 5035 scope.go:122] "RemoveContainer" containerID="d1af9cc72a9a0255020e279ffaa2cdd2dbabf4835950a9d676e310431ce30975" Oct 30 02:40:15.473326 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:15.473278 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv"] Oct 30 02:40:15.481957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:15.481920 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-93ec46-7bb6f559fd-z2ncv"] Oct 30 02:40:15.644678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:15.644645 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c0e8297-c571-4105-b69a-c7eb9c3938be" path="/var/lib/kubelet/pods/6c0e8297-c571-4105-b69a-c7eb9c3938be/volumes" Oct 30 02:40:17.743499 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:40:17.743453 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:40:20.430013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:20.429975 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" podUID="6c9850ac-3004-42da-8ea5-d005e5f5bbca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:20.430424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:20.430045 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" podUID="b6267677-8af0-44d5-9d95-8e8eb629463e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:30.429941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:30.429898 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" podUID="6c9850ac-3004-42da-8ea5-d005e5f5bbca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:30.430383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:30.429896 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" podUID="b6267677-8af0-44d5-9d95-8e8eb629463e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:40:31.587198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.587159 5035 generic.go:356] "Generic (PLEG): container finished" podID="6c9850ac-3004-42da-8ea5-d005e5f5bbca" containerID="6d41cf34e8f4c2974bb18ffe21197f8d96bc27ecc4104e440f26f4edee183c26" exitCode=0 Oct 30 02:40:31.587619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.587205 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" event={"ID":"6c9850ac-3004-42da-8ea5-d005e5f5bbca","Type":"ContainerDied","Data":"6d41cf34e8f4c2974bb18ffe21197f8d96bc27ecc4104e440f26f4edee183c26"} Oct 30 02:40:31.589391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.589353 5035 generic.go:356] "Generic (PLEG): container finished" podID="b6267677-8af0-44d5-9d95-8e8eb629463e" containerID="5bc0bad9a453c4a2ac671e105a4c8073503ff578b66ebec6a26086c49648e3c6" exitCode=0 Oct 30 02:40:31.589565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.589541 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" event={"ID":"b6267677-8af0-44d5-9d95-8e8eb629463e","Type":"ContainerDied","Data":"5bc0bad9a453c4a2ac671e105a4c8073503ff578b66ebec6a26086c49648e3c6"} Oct 30 02:40:31.780199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.780171 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:40:31.805774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.805744 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:40:31.869547 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.867516 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6267677-8af0-44d5-9d95-8e8eb629463e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6267677-8af0-44d5-9d95-8e8eb629463e-plugins-config-volume\") pod \"b6267677-8af0-44d5-9d95-8e8eb629463e\" (UID: \"b6267677-8af0-44d5-9d95-8e8eb629463e\") " Oct 30 02:40:31.869547 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.867723 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6267677-8af0-44d5-9d95-8e8eb629463e-kube-api-access-9w82m\" (UniqueName: \"kubernetes.io/projected/b6267677-8af0-44d5-9d95-8e8eb629463e-kube-api-access-9w82m\") pod \"b6267677-8af0-44d5-9d95-8e8eb629463e\" (UID: \"b6267677-8af0-44d5-9d95-8e8eb629463e\") " Oct 30 02:40:31.869547 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.869015 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6267677-8af0-44d5-9d95-8e8eb629463e-plugins-config-volume" pod "b6267677-8af0-44d5-9d95-8e8eb629463e" (UID: "b6267677-8af0-44d5-9d95-8e8eb629463e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:31.872658 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.872624 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6267677-8af0-44d5-9d95-8e8eb629463e-kube-api-access-9w82m" pod "b6267677-8af0-44d5-9d95-8e8eb629463e" (UID: "b6267677-8af0-44d5-9d95-8e8eb629463e"). InnerVolumeSpecName "kube-api-access-9w82m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:31.969596 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.969504 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9850ac-3004-42da-8ea5-d005e5f5bbca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9850ac-3004-42da-8ea5-d005e5f5bbca-plugins-config-volume\") pod \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\" (UID: \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\") " Oct 30 02:40:31.969764 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.969705 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9850ac-3004-42da-8ea5-d005e5f5bbca-kube-api-access-8rpqw\" (UniqueName: \"kubernetes.io/projected/6c9850ac-3004-42da-8ea5-d005e5f5bbca-kube-api-access-8rpqw\") pod \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\" (UID: \"6c9850ac-3004-42da-8ea5-d005e5f5bbca\") " Oct 30 02:40:31.969869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.969847 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9850ac-3004-42da-8ea5-d005e5f5bbca-plugins-config-volume" pod "6c9850ac-3004-42da-8ea5-d005e5f5bbca" (UID: "6c9850ac-3004-42da-8ea5-d005e5f5bbca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:31.970265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.970236 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w82m\" (UniqueName: \"kubernetes.io/projected/b6267677-8af0-44d5-9d95-8e8eb629463e-kube-api-access-9w82m\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:31.970392 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.970269 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9850ac-3004-42da-8ea5-d005e5f5bbca-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:31.970392 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.970282 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6267677-8af0-44d5-9d95-8e8eb629463e-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:31.971907 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:31.971884 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9850ac-3004-42da-8ea5-d005e5f5bbca-kube-api-access-8rpqw" pod "6c9850ac-3004-42da-8ea5-d005e5f5bbca" (UID: "6c9850ac-3004-42da-8ea5-d005e5f5bbca"). InnerVolumeSpecName "kube-api-access-8rpqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:32.072096 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.072053 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rpqw\" (UniqueName: \"kubernetes.io/projected/6c9850ac-3004-42da-8ea5-d005e5f5bbca-kube-api-access-8rpqw\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:32.599323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.599281 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" Oct 30 02:40:32.599859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.599288 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r" event={"ID":"b6267677-8af0-44d5-9d95-8e8eb629463e","Type":"ContainerDied","Data":"87d62d66af537d3d072b0ce2710d8eb4d2bf2d6aaf9227e2f4be456188d7c43e"} Oct 30 02:40:32.599859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.599465 5035 scope.go:122] "RemoveContainer" containerID="5bc0bad9a453c4a2ac671e105a4c8073503ff578b66ebec6a26086c49648e3c6" Oct 30 02:40:32.601078 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.601050 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" event={"ID":"6c9850ac-3004-42da-8ea5-d005e5f5bbca","Type":"ContainerDied","Data":"8a3ff9c5f210ad7f700ac892e29ff2867573437a8f1b671475c366eb3f78048b"} Oct 30 02:40:32.601208 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.601100 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn" Oct 30 02:40:32.612762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.612739 5035 scope.go:122] "RemoveContainer" containerID="6d41cf34e8f4c2974bb18ffe21197f8d96bc27ecc4104e440f26f4edee183c26" Oct 30 02:40:32.660453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.660417 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn"] Oct 30 02:40:32.671893 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.671858 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4153e5-547d45cf79-fhshn"] Oct 30 02:40:32.716664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.716563 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r"] Oct 30 02:40:32.722461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:32.722429 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4153e5-b59c4f658-tgf8r"] Oct 30 02:40:33.646513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:33.646471 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9850ac-3004-42da-8ea5-d005e5f5bbca" path="/var/lib/kubelet/pods/6c9850ac-3004-42da-8ea5-d005e5f5bbca/volumes" Oct 30 02:40:33.647246 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:33.647223 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6267677-8af0-44d5-9d95-8e8eb629463e" path="/var/lib/kubelet/pods/b6267677-8af0-44d5-9d95-8e8eb629463e/volumes" Oct 30 02:40:47.629747 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:40:47.629708 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:40:59.402213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.402172 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4"] Oct 30 02:40:59.402825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.402422 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" podUID="46681de3-fb8d-498e-a12f-2a2eee5f5a40" containerName="maas-api" containerID="cri-o://3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c" gracePeriod=30 Oct 30 02:40:59.726014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.725984 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:40:59.807676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.807626 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46681de3-fb8d-498e-a12f-2a2eee5f5a40-kube-api-access-7r64j\" (UniqueName: \"kubernetes.io/projected/46681de3-fb8d-498e-a12f-2a2eee5f5a40-kube-api-access-7r64j\") pod \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " Oct 30 02:40:59.807925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.807802 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46681de3-fb8d-498e-a12f-2a2eee5f5a40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46681de3-fb8d-498e-a12f-2a2eee5f5a40-openshift-service-ca\") pod \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " Oct 30 02:40:59.808007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.807986 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-metrics-certs\") pod \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " Oct 30 02:40:59.808099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.808079 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-maas-api-tls\") pod \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\" (UID: \"46681de3-fb8d-498e-a12f-2a2eee5f5a40\") " Oct 30 02:40:59.809274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.809237 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46681de3-fb8d-498e-a12f-2a2eee5f5a40-openshift-service-ca" pod "46681de3-fb8d-498e-a12f-2a2eee5f5a40" (UID: "46681de3-fb8d-498e-a12f-2a2eee5f5a40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:59.812674 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.812626 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46681de3-fb8d-498e-a12f-2a2eee5f5a40-kube-api-access-7r64j" pod "46681de3-fb8d-498e-a12f-2a2eee5f5a40" (UID: "46681de3-fb8d-498e-a12f-2a2eee5f5a40"). InnerVolumeSpecName "kube-api-access-7r64j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:59.827689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.827001 5035 generic.go:356] "Generic (PLEG): container finished" podID="46681de3-fb8d-498e-a12f-2a2eee5f5a40" containerID="3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c" exitCode=0 Oct 30 02:40:59.827689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.827248 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" event={"ID":"46681de3-fb8d-498e-a12f-2a2eee5f5a40","Type":"ContainerDied","Data":"3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c"} Oct 30 02:40:59.827689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.827303 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" event={"ID":"46681de3-fb8d-498e-a12f-2a2eee5f5a40","Type":"ContainerDied","Data":"6d2137e6cd1fd84833e1ba7b59b0da5fcaa7d3920d7c11376dc1dc8e91a2a9c3"} Oct 30 02:40:59.827689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.827334 5035 scope.go:122] "RemoveContainer" containerID="3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c" Oct 30 02:40:59.829726 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.828242 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4" Oct 30 02:40:59.838843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.838797 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-maas-api-tls" pod "46681de3-fb8d-498e-a12f-2a2eee5f5a40" (UID: "46681de3-fb8d-498e-a12f-2a2eee5f5a40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:40:59.840521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.840090 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-metrics-certs" pod "46681de3-fb8d-498e-a12f-2a2eee5f5a40" (UID: "46681de3-fb8d-498e-a12f-2a2eee5f5a40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:40:59.877041 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.877004 5035 scope.go:122] "RemoveContainer" containerID="3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c" Oct 30 02:40:59.877477 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:40:59.877445 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c\": container with ID starting with 3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c not found: ID does not exist" containerID="3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c" Oct 30 02:40:59.877612 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.877492 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c"} err="failed to get container status \"3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c\": rpc error: code = NotFound desc = could not find container \"3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c\": container with ID starting with 3dc8dc69356fa9031e9ac03346dec92f8ec20f21c97779ed43fea9f1249b047c not found: ID does not exist" Oct 30 02:40:59.909886 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.909845 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:59.909886 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.909885 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46681de3-fb8d-498e-a12f-2a2eee5f5a40-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:59.910131 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.909897 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r64j\" (UniqueName: \"kubernetes.io/projected/46681de3-fb8d-498e-a12f-2a2eee5f5a40-kube-api-access-7r64j\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:40:59.910131 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:40:59.909907 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46681de3-fb8d-498e-a12f-2a2eee5f5a40-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:00.204821 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:00.204776 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4"] Oct 30 02:41:00.205120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:00.205104 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0ce950-75c569d7-shfv4"] Oct 30 02:41:01.644612 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:01.644569 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46681de3-fb8d-498e-a12f-2a2eee5f5a40" path="/var/lib/kubelet/pods/46681de3-fb8d-498e-a12f-2a2eee5f5a40/volumes" Oct 30 02:41:11.927760 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:11.927721 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n"] Oct 30 02:41:11.928495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:11.928018 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" podUID="e63eff88-dd24-4f72-ac4b-76f163edd204" containerName="maas-api" containerID="cri-o://7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344" gracePeriod=30 Oct 30 02:41:12.255791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.255763 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:41:12.386107 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.386054 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63eff88-dd24-4f72-ac4b-76f163edd204-kube-api-access-s96f7\" (UniqueName: \"kubernetes.io/projected/e63eff88-dd24-4f72-ac4b-76f163edd204-kube-api-access-s96f7\") pod \"e63eff88-dd24-4f72-ac4b-76f163edd204\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " Oct 30 02:41:12.386315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.386296 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-maas-api-tls\") pod \"e63eff88-dd24-4f72-ac4b-76f163edd204\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " Oct 30 02:41:12.386426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.386408 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63eff88-dd24-4f72-ac4b-76f163edd204-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63eff88-dd24-4f72-ac4b-76f163edd204-openshift-service-ca\") pod \"e63eff88-dd24-4f72-ac4b-76f163edd204\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " Oct 30 02:41:12.386539 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.386446 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-metrics-certs\") pod \"e63eff88-dd24-4f72-ac4b-76f163edd204\" (UID: \"e63eff88-dd24-4f72-ac4b-76f163edd204\") " Oct 30 02:41:12.387040 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.386987 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63eff88-dd24-4f72-ac4b-76f163edd204-openshift-service-ca" pod "e63eff88-dd24-4f72-ac4b-76f163edd204" (UID: "e63eff88-dd24-4f72-ac4b-76f163edd204"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:12.388982 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.388948 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-maas-api-tls" pod "e63eff88-dd24-4f72-ac4b-76f163edd204" (UID: "e63eff88-dd24-4f72-ac4b-76f163edd204"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:12.389117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.389078 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63eff88-dd24-4f72-ac4b-76f163edd204-kube-api-access-s96f7" pod "e63eff88-dd24-4f72-ac4b-76f163edd204" (UID: "e63eff88-dd24-4f72-ac4b-76f163edd204"). InnerVolumeSpecName "kube-api-access-s96f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:12.390071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.389878 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-metrics-certs" pod "e63eff88-dd24-4f72-ac4b-76f163edd204" (UID: "e63eff88-dd24-4f72-ac4b-76f163edd204"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:12.488244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.488137 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63eff88-dd24-4f72-ac4b-76f163edd204-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:12.488244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.488176 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:12.488244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.488190 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s96f7\" (UniqueName: \"kubernetes.io/projected/e63eff88-dd24-4f72-ac4b-76f163edd204-kube-api-access-s96f7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:12.488244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.488203 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63eff88-dd24-4f72-ac4b-76f163edd204-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:12.940189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.940093 5035 generic.go:356] "Generic (PLEG): container finished" podID="e63eff88-dd24-4f72-ac4b-76f163edd204" containerID="7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344" exitCode=0 Oct 30 02:41:12.940189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.940143 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" Oct 30 02:41:12.940189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.940161 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" event={"ID":"e63eff88-dd24-4f72-ac4b-76f163edd204","Type":"ContainerDied","Data":"7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344"} Oct 30 02:41:12.940781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.940220 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n" event={"ID":"e63eff88-dd24-4f72-ac4b-76f163edd204","Type":"ContainerDied","Data":"6e66bea45dd6aec02335b25ad4e11bc0038076a291d9f4e49d8e00985fb1df17"} Oct 30 02:41:12.940781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.940243 5035 scope.go:122] "RemoveContainer" containerID="7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344" Oct 30 02:41:12.951846 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.951822 5035 scope.go:122] "RemoveContainer" containerID="7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344" Oct 30 02:41:12.952128 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:41:12.952110 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344\": container with ID starting with 7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344 not found: ID does not exist" containerID="7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344" Oct 30 02:41:12.952184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:12.952142 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344"} err="failed to get container status \"7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344\": rpc error: code = NotFound desc = could not find container \"7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344\": container with ID starting with 7850f0b093f69a4760b6e6250c69ba98f4ca3ed687f73cd559006d4246051344 not found: ID does not exist" Oct 30 02:41:13.001013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:13.000973 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n"] Oct 30 02:41:13.012486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:13.012451 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-17e747-86bb778b47-j9w2n"] Oct 30 02:41:13.087616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:13.087551 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l"] Oct 30 02:41:13.187165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:13.187121 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq"] Oct 30 02:41:13.647590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:13.647512 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63eff88-dd24-4f72-ac4b-76f163edd204" path="/var/lib/kubelet/pods/e63eff88-dd24-4f72-ac4b-76f163edd204/volumes" Oct 30 02:41:17.630709 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:41:17.630668 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:41:18.088006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:18.087933 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" podUID="0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" containerName="payload-processing" containerID="cri-o://df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116" gracePeriod=25 Oct 30 02:41:18.190268 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:18.190213 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" podUID="59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" containerName="payload-pre-processing" containerID="cri-o://818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee" gracePeriod=25 Oct 30 02:41:22.427799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:22.427754 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2"] Oct 30 02:41:22.428342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:22.428042 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" containerID="cri-o://b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019" gracePeriod=30 Oct 30 02:41:23.713891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:23.713788 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" podUID="59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:23.714355 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:23.713938 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" podUID="0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:27.769055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.769022 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:41:27.917094 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.917048 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4"] Oct 30 02:41:27.921991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.921961 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-envoy\") pod \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " Oct 30 02:41:27.922210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922035 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-kube-api-access-7lcln\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-kube-api-access-7lcln\") pod \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " Oct 30 02:41:27.922210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922142 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-credential-socket\") pod \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " Oct 30 02:41:27.922210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922198 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-certs\") pod \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " Oct 30 02:41:27.922436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922302 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-socket\") pod \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " Oct 30 02:41:27.922436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922395 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-token\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-token\") pod \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " Oct 30 02:41:27.922625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922459 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-podinfo\") pod \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " Oct 30 02:41:27.922625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922459 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-credential-socket" pod "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" (UID: "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:27.922625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922489 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istiod-ca-cert\") pod \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " Oct 30 02:41:27.922625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922608 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-data\") pod \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\" (UID: \"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3\") " Oct 30 02:41:27.922625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922593 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-certs" pod "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" (UID: "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:27.922981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922961 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-socket" pod "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" (UID: "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:27.923040 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.922985 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istiod-ca-cert" pod "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" (UID: "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:27.923335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.923194 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-data" pod "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" (UID: "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:27.923335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.923247 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:27.923335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.923261 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:27.923335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.923270 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:27.923335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.923288 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:27.925429 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.925400 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-token" pod "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" (UID: "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:27.925805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.925772 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-podinfo" pod "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" (UID: "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:41:27.925906 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.925809 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-envoy" pod "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" (UID: "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:27.927257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.927227 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb"] Oct 30 02:41:27.927579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.927516 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-kube-api-access-7lcln" pod "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" (UID: "6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3"). InnerVolumeSpecName "kube-api-access-7lcln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:27.931787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:27.931758 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw"] Oct 30 02:41:28.024467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.024371 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:28.024467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.024405 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:28.024467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.024414 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:28.024467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.024425 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:28.024467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.024434 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lcln\" (UniqueName: \"kubernetes.io/projected/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3-kube-api-access-7lcln\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:28.080619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.080582 5035 generic.go:356] "Generic (PLEG): container finished" podID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerID="b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019" exitCode=0 Oct 30 02:41:28.080820 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.080646 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" Oct 30 02:41:28.080820 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.080665 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" event={"ID":"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3","Type":"ContainerDied","Data":"b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019"} Oct 30 02:41:28.080820 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.080744 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2" event={"ID":"6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3","Type":"ContainerDied","Data":"fafa363fdda5472506ae7dfa74fba48a7c863cd5274d3d9d1733577d8162e86a"} Oct 30 02:41:28.080820 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.080768 5035 scope.go:122] "RemoveContainer" containerID="b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019" Oct 30 02:41:28.094584 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.094559 5035 scope.go:122] "RemoveContainer" containerID="5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5" Oct 30 02:41:28.105146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.105122 5035 scope.go:122] "RemoveContainer" containerID="b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019" Oct 30 02:41:28.105453 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:41:28.105428 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019\": container with ID starting with b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019 not found: ID does not exist" containerID="b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019" Oct 30 02:41:28.105576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.105464 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019"} err="failed to get container status \"b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019\": rpc error: code = NotFound desc = could not find container \"b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019\": container with ID starting with b1c3f71afcac58283c50801e3eb5de3584149a4c50e37989019379ecd7d33019 not found: ID does not exist" Oct 30 02:41:28.105576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.105482 5035 scope.go:122] "RemoveContainer" containerID="5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5" Oct 30 02:41:28.105818 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:41:28.105798 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5\": container with ID starting with 5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5 not found: ID does not exist" containerID="5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5" Oct 30 02:41:28.105869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.105824 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5"} err="failed to get container status \"5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5\": rpc error: code = NotFound desc = could not find container \"5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5\": container with ID starting with 5f54af5f0253d228dc20ee9d85e86895306d75a50b0d6145ad4cfef5157da5d5 not found: ID does not exist" Oct 30 02:41:28.137099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.137061 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2"] Oct 30 02:41:28.140841 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:28.140804 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-17e747-openshift-default-6bfc467795-xcsj2"] Oct 30 02:41:29.645200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:29.645165 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" path="/var/lib/kubelet/pods/6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3/volumes" Oct 30 02:41:33.713050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:33.713007 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" podUID="59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:33.713600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:33.713007 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" podUID="0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:41:38.387012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.386962 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:41:38.484019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.483990 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-kube-api-access-j9rzn\" (UniqueName: \"kubernetes.io/projected/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-kube-api-access-j9rzn\") pod \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\" (UID: \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\") " Oct 30 02:41:38.484217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.484161 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-plugins-config-volume\") pod \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\" (UID: \"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf\") " Oct 30 02:41:38.484545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.484493 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-plugins-config-volume" pod "0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" (UID: "0632efb9-a10a-4f96-8261-2b6ca3e1ffdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:38.486483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.486452 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-kube-api-access-j9rzn" pod "0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" (UID: "0632efb9-a10a-4f96-8261-2b6ca3e1ffdf"). InnerVolumeSpecName "kube-api-access-j9rzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:38.488668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.488650 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:41:38.585938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.585900 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-plugins-config-volume\") pod \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\" (UID: \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\") " Oct 30 02:41:38.586123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.586100 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-kube-api-access-bvw2t\" (UniqueName: \"kubernetes.io/projected/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-kube-api-access-bvw2t\") pod \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\" (UID: \"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0\") " Oct 30 02:41:38.586343 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.586317 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-plugins-config-volume" pod "59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" (UID: "59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:38.586744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.586721 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rzn\" (UniqueName: \"kubernetes.io/projected/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-kube-api-access-j9rzn\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:38.586744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.586744 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:38.586920 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.586753 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:38.588266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.588243 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-kube-api-access-bvw2t" pod "59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" (UID: "59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0"). InnerVolumeSpecName "kube-api-access-bvw2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:38.687865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:38.687763 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvw2t\" (UniqueName: \"kubernetes.io/projected/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0-kube-api-access-bvw2t\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:39.166009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.165969 5035 generic.go:356] "Generic (PLEG): container finished" podID="0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" containerID="df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116" exitCode=0 Oct 30 02:41:39.166214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.166029 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" Oct 30 02:41:39.166214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.166049 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" event={"ID":"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf","Type":"ContainerDied","Data":"df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116"} Oct 30 02:41:39.166214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.166127 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l" event={"ID":"0632efb9-a10a-4f96-8261-2b6ca3e1ffdf","Type":"ContainerDied","Data":"df1f4387f3cee0aab3bda8ca78b1cb094c7997ae0ec079351503da229f74d22b"} Oct 30 02:41:39.166214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.166148 5035 scope.go:122] "RemoveContainer" containerID="df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116" Oct 30 02:41:39.167719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.167698 5035 generic.go:356] "Generic (PLEG): container finished" podID="59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" containerID="818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee" exitCode=0 Oct 30 02:41:39.167824 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.167732 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" event={"ID":"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0","Type":"ContainerDied","Data":"818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee"} Oct 30 02:41:39.167824 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.167756 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" Oct 30 02:41:39.167824 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.167787 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq" event={"ID":"59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0","Type":"ContainerDied","Data":"08d28101a5116481b15896c9ba399222dafdf1a8b2993c85663a41386660cea0"} Oct 30 02:41:39.177354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.177326 5035 scope.go:122] "RemoveContainer" containerID="df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116" Oct 30 02:41:39.177725 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:41:39.177705 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116\": container with ID starting with df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116 not found: ID does not exist" containerID="df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116" Oct 30 02:41:39.177824 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.177732 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116"} err="failed to get container status \"df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116\": rpc error: code = NotFound desc = could not find container \"df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116\": container with ID starting with df1f04c73ebc1939145d2678c92b023da3d4af174a422b018a68a80637ca3116 not found: ID does not exist" Oct 30 02:41:39.177824 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.177763 5035 scope.go:122] "RemoveContainer" containerID="818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee" Oct 30 02:41:39.189937 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.189916 5035 scope.go:122] "RemoveContainer" containerID="818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee" Oct 30 02:41:39.190275 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:41:39.190249 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee\": container with ID starting with 818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee not found: ID does not exist" containerID="818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee" Oct 30 02:41:39.190333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.190285 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee"} err="failed to get container status \"818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee\": rpc error: code = NotFound desc = could not find container \"818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee\": container with ID starting with 818f460bd248bc396389bf1d63fbcdb80c91e7458643778b6a4e73b8bd926eee not found: ID does not exist" Oct 30 02:41:39.220164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.220126 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq"] Oct 30 02:41:39.245679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.244882 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-000edd-d5b7c669f-m9hfq"] Oct 30 02:41:39.265264 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.265224 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l"] Oct 30 02:41:39.268555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.268512 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-000edd-64d8869796-mcg8l"] Oct 30 02:41:39.644680 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.644640 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" path="/var/lib/kubelet/pods/0632efb9-a10a-4f96-8261-2b6ca3e1ffdf/volumes" Oct 30 02:41:39.645207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:39.645191 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" path="/var/lib/kubelet/pods/59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0/volumes" Oct 30 02:41:42.943453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:42.943404 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" podUID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" containerName="main" containerID="cri-o://2ea917afb0365425834fbdf7b64c52bc2cd22cf93e841e8936dbb36aaabad861" gracePeriod=45 Oct 30 02:41:42.950225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:42.950186 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" containerID="cri-o://73098f44d1f6ff089d382c4d6036ed1fe94f5447c99e1f0ff3cd3b6080816ad3" gracePeriod=45 Oct 30 02:41:42.954355 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:42.954320 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" containerID="cri-o://4a60d10c03adeb9b8b4c78761e2dedb503f90e9b982863df66f1799b6db77b4c" gracePeriod=45 Oct 30 02:41:42.982895 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:42.982857 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": dial tcp 10.132.0.90:8000: connect: connection refused" Oct 30 02:41:43.204831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.204648 5035 generic.go:356] "Generic (PLEG): container finished" podID="0590d679-8944-4559-a287-07a90e6d0f42" containerID="4a60d10c03adeb9b8b4c78761e2dedb503f90e9b982863df66f1799b6db77b4c" exitCode=0 Oct 30 02:41:43.204831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.204765 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" event={"ID":"0590d679-8944-4559-a287-07a90e6d0f42","Type":"ContainerDied","Data":"4a60d10c03adeb9b8b4c78761e2dedb503f90e9b982863df66f1799b6db77b4c"} Oct 30 02:41:43.261283 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.261245 5035 generic.go:356] "Generic (PLEG): container finished" podID="df753c92-98ff-4ceb-8468-6766ded63515" containerID="73098f44d1f6ff089d382c4d6036ed1fe94f5447c99e1f0ff3cd3b6080816ad3" exitCode=0 Oct 30 02:41:43.263590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.261708 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" event={"ID":"df753c92-98ff-4ceb-8468-6766ded63515","Type":"ContainerDied","Data":"73098f44d1f6ff089d382c4d6036ed1fe94f5447c99e1f0ff3cd3b6080816ad3"} Oct 30 02:41:43.267811 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.267367 5035 generic.go:356] "Generic (PLEG): container finished" podID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" containerID="2ea917afb0365425834fbdf7b64c52bc2cd22cf93e841e8936dbb36aaabad861" exitCode=0 Oct 30 02:41:43.267811 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.267658 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" event={"ID":"1a1a7bde-671c-4b7f-96d9-a79e994f56c3","Type":"ContainerDied","Data":"2ea917afb0365425834fbdf7b64c52bc2cd22cf93e841e8936dbb36aaabad861"} Oct 30 02:41:43.308301 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.308269 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 02:41:43.313493 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.313458 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 02:41:43.317911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.317873 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 02:41:43.364308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364276 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-dshm\") pod \"df753c92-98ff-4ceb-8468-6766ded63515\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " Oct 30 02:41:43.364308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364314 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-dshm\") pod \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " Oct 30 02:41:43.364623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364382 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-model-cache\") pod \"0590d679-8944-4559-a287-07a90e6d0f42\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " Oct 30 02:41:43.364623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364444 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tls-certs\") pod \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " Oct 30 02:41:43.364623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364472 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0590d679-8944-4559-a287-07a90e6d0f42-kube-api-access-22klc\" (UniqueName: \"kubernetes.io/projected/0590d679-8944-4559-a287-07a90e6d0f42-kube-api-access-22klc\") pod \"0590d679-8944-4559-a287-07a90e6d0f42\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " Oct 30 02:41:43.364623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364492 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-tmp-dir\") pod \"0590d679-8944-4559-a287-07a90e6d0f42\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " Oct 30 02:41:43.364623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364568 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df753c92-98ff-4ceb-8468-6766ded63515-tls-certs\" (UniqueName: \"kubernetes.io/secret/df753c92-98ff-4ceb-8468-6766ded63515-tls-certs\") pod \"df753c92-98ff-4ceb-8468-6766ded63515\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " Oct 30 02:41:43.364623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364599 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-kube-api-access-n46xq\" (UniqueName: \"kubernetes.io/projected/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-kube-api-access-n46xq\") pod \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " Oct 30 02:41:43.364976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364645 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0590d679-8944-4559-a287-07a90e6d0f42-tls-certs\" (UniqueName: \"kubernetes.io/secret/0590d679-8944-4559-a287-07a90e6d0f42-tls-certs\") pod \"0590d679-8944-4559-a287-07a90e6d0f42\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " Oct 30 02:41:43.364976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364690 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-model-cache\") pod \"df753c92-98ff-4ceb-8468-6766ded63515\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " Oct 30 02:41:43.364976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364763 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-home\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-home\") pod \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " Oct 30 02:41:43.364976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364800 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-home\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-home\") pod \"0590d679-8944-4559-a287-07a90e6d0f42\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " Oct 30 02:41:43.364976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364850 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-home\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-home\") pod \"df753c92-98ff-4ceb-8468-6766ded63515\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " Oct 30 02:41:43.364976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364900 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-tmp-dir\") pod \"df753c92-98ff-4ceb-8468-6766ded63515\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " Oct 30 02:41:43.364976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364957 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-model-cache\") pod \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " Oct 30 02:41:43.365342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364996 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-dshm\") pod \"0590d679-8944-4559-a287-07a90e6d0f42\" (UID: \"0590d679-8944-4559-a287-07a90e6d0f42\") " Oct 30 02:41:43.365342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.364983 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-model-cache" pod "0590d679-8944-4559-a287-07a90e6d0f42" (UID: "0590d679-8944-4559-a287-07a90e6d0f42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.365342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.365156 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tmp-dir\") pod \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\" (UID: \"1a1a7bde-671c-4b7f-96d9-a79e994f56c3\") " Oct 30 02:41:43.365342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.365202 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df753c92-98ff-4ceb-8468-6766ded63515-kube-api-access-cfznq\" (UniqueName: \"kubernetes.io/projected/df753c92-98ff-4ceb-8468-6766ded63515-kube-api-access-cfznq\") pod \"df753c92-98ff-4ceb-8468-6766ded63515\" (UID: \"df753c92-98ff-4ceb-8468-6766ded63515\") " Oct 30 02:41:43.365890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.365862 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.365890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.365198 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-home" pod "0590d679-8944-4559-a287-07a90e6d0f42" (UID: "0590d679-8944-4559-a287-07a90e6d0f42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.366209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.365504 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-tmp-dir" pod "0590d679-8944-4559-a287-07a90e6d0f42" (UID: "0590d679-8944-4559-a287-07a90e6d0f42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.366209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.366139 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-home" pod "df753c92-98ff-4ceb-8468-6766ded63515" (UID: "df753c92-98ff-4ceb-8468-6766ded63515"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.366374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.366352 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-tmp-dir" pod "df753c92-98ff-4ceb-8468-6766ded63515" (UID: "df753c92-98ff-4ceb-8468-6766ded63515"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.366583 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.366558 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-model-cache" pod "1a1a7bde-671c-4b7f-96d9-a79e994f56c3" (UID: "1a1a7bde-671c-4b7f-96d9-a79e994f56c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.367643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.367612 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-kube-api-access-n46xq" pod "1a1a7bde-671c-4b7f-96d9-a79e994f56c3" (UID: "1a1a7bde-671c-4b7f-96d9-a79e994f56c3"). InnerVolumeSpecName "kube-api-access-n46xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:43.368165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.367953 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0590d679-8944-4559-a287-07a90e6d0f42-kube-api-access-22klc" pod "0590d679-8944-4559-a287-07a90e6d0f42" (UID: "0590d679-8944-4559-a287-07a90e6d0f42"). InnerVolumeSpecName "kube-api-access-22klc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:43.368473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.368312 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tmp-dir" pod "1a1a7bde-671c-4b7f-96d9-a79e994f56c3" (UID: "1a1a7bde-671c-4b7f-96d9-a79e994f56c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.368761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.368681 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-model-cache" pod "df753c92-98ff-4ceb-8468-6766ded63515" (UID: "df753c92-98ff-4ceb-8468-6766ded63515"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.368761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.368687 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df753c92-98ff-4ceb-8468-6766ded63515-tls-certs" pod "df753c92-98ff-4ceb-8468-6766ded63515" (UID: "df753c92-98ff-4ceb-8468-6766ded63515"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:43.368761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.368667 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-dshm" pod "df753c92-98ff-4ceb-8468-6766ded63515" (UID: "df753c92-98ff-4ceb-8468-6766ded63515"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.369001 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.368907 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-home" pod "1a1a7bde-671c-4b7f-96d9-a79e994f56c3" (UID: "1a1a7bde-671c-4b7f-96d9-a79e994f56c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.369160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.369134 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0590d679-8944-4559-a287-07a90e6d0f42-tls-certs" pod "0590d679-8944-4559-a287-07a90e6d0f42" (UID: "0590d679-8944-4559-a287-07a90e6d0f42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:43.369284 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.369259 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-dshm" pod "0590d679-8944-4559-a287-07a90e6d0f42" (UID: "0590d679-8944-4559-a287-07a90e6d0f42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.369365 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.369331 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tls-certs" pod "1a1a7bde-671c-4b7f-96d9-a79e994f56c3" (UID: "1a1a7bde-671c-4b7f-96d9-a79e994f56c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:43.370258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.370227 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-dshm" pod "1a1a7bde-671c-4b7f-96d9-a79e994f56c3" (UID: "1a1a7bde-671c-4b7f-96d9-a79e994f56c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.370625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.370607 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df753c92-98ff-4ceb-8468-6766ded63515-kube-api-access-cfznq" pod "df753c92-98ff-4ceb-8468-6766ded63515" (UID: "df753c92-98ff-4ceb-8468-6766ded63515"). InnerVolumeSpecName "kube-api-access-cfznq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:43.468104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468066 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468107 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22klc\" (UniqueName: \"kubernetes.io/projected/0590d679-8944-4559-a287-07a90e6d0f42-kube-api-access-22klc\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468165 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468182 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df753c92-98ff-4ceb-8468-6766ded63515-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468195 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n46xq\" (UniqueName: \"kubernetes.io/projected/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-kube-api-access-n46xq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468208 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0590d679-8944-4559-a287-07a90e6d0f42-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468222 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468254 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468268 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468279 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468292 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468306 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468319 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0590d679-8944-4559-a287-07a90e6d0f42-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468330 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468345 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfznq\" (UniqueName: \"kubernetes.io/projected/df753c92-98ff-4ceb-8468-6766ded63515-kube-api-access-cfznq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468357 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df753c92-98ff-4ceb-8468-6766ded63515-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.468918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.468369 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1a7bde-671c-4b7f-96d9-a79e994f56c3-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.721492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.721391 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb"] Oct 30 02:41:43.721725 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:43.721674 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" podUID="1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" containerName="maas-api" containerID="cri-o://f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91" gracePeriod=30 Oct 30 02:41:44.038616 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.038585 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:41:44.081497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.081457 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-maas-api-tls\") pod \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " Oct 30 02:41:44.081755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.081520 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-openshift-service-ca\") pod \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " Oct 30 02:41:44.081755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.081601 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-metrics-certs\") pod \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " Oct 30 02:41:44.081755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.081659 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-kube-api-access-nzr8b\" (UniqueName: \"kubernetes.io/projected/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-kube-api-access-nzr8b\") pod \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\" (UID: \"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993\") " Oct 30 02:41:44.082033 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.081999 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-openshift-service-ca" pod "1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" (UID: "1dab5d1f-f9ea-4b1f-82cf-97ebbd947993"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:44.082460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.082436 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:44.084651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.084499 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-metrics-certs" pod "1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" (UID: "1dab5d1f-f9ea-4b1f-82cf-97ebbd947993"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:44.084771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.084694 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-kube-api-access-nzr8b" pod "1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" (UID: "1dab5d1f-f9ea-4b1f-82cf-97ebbd947993"). InnerVolumeSpecName "kube-api-access-nzr8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:44.084955 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.084935 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-maas-api-tls" pod "1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" (UID: "1dab5d1f-f9ea-4b1f-82cf-97ebbd947993"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:44.184330 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.184288 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:44.184330 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.184325 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzr8b\" (UniqueName: \"kubernetes.io/projected/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-kube-api-access-nzr8b\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:44.184330 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.184335 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:41:44.278464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.278381 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" Oct 30 02:41:44.278464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.278410 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw" event={"ID":"0590d679-8944-4559-a287-07a90e6d0f42","Type":"ContainerDied","Data":"5fffa7e944c121b2cf7294d37a4ed3f58ccf21e0ca39ee0aa92fa084b619d3e7"} Oct 30 02:41:44.278760 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.278471 5035 scope.go:122] "RemoveContainer" containerID="4a60d10c03adeb9b8b4c78761e2dedb503f90e9b982863df66f1799b6db77b4c" Oct 30 02:41:44.280135 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.280114 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" Oct 30 02:41:44.280252 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.280129 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb" event={"ID":"df753c92-98ff-4ceb-8468-6766ded63515","Type":"ContainerDied","Data":"39ac6f7d1adb4a336d1a7b919f158fe79665b30eae6609add10598016577f1b7"} Oct 30 02:41:44.281803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.281780 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" event={"ID":"1a1a7bde-671c-4b7f-96d9-a79e994f56c3","Type":"ContainerDied","Data":"9057d6ee189374b5c4a0b9dbba96bdd92e71430d1d25d7f39f43614719e296ff"} Oct 30 02:41:44.281894 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.281801 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4" Oct 30 02:41:44.283828 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.283803 5035 generic.go:356] "Generic (PLEG): container finished" podID="1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" containerID="f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91" exitCode=0 Oct 30 02:41:44.283933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.283837 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" event={"ID":"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993","Type":"ContainerDied","Data":"f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91"} Oct 30 02:41:44.283933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.283844 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" Oct 30 02:41:44.283933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.283922 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb" event={"ID":"1dab5d1f-f9ea-4b1f-82cf-97ebbd947993","Type":"ContainerDied","Data":"b882acd11cbe70792d281f1b0dfb27fd850cef6cc7ed6514ba4b5fcce340b8d2"} Oct 30 02:41:44.291327 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.291298 5035 scope.go:122] "RemoveContainer" containerID="73098f44d1f6ff089d382c4d6036ed1fe94f5447c99e1f0ff3cd3b6080816ad3" Oct 30 02:41:44.302033 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.302010 5035 scope.go:122] "RemoveContainer" containerID="2ea917afb0365425834fbdf7b64c52bc2cd22cf93e841e8936dbb36aaabad861" Oct 30 02:41:44.312767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.312746 5035 scope.go:122] "RemoveContainer" containerID="f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91" Oct 30 02:41:44.322901 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.322877 5035 scope.go:122] "RemoveContainer" containerID="f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91" Oct 30 02:41:44.323231 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:41:44.323210 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91\": container with ID starting with f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91 not found: ID does not exist" containerID="f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91" Oct 30 02:41:44.323292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.323240 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91"} err="failed to get container status \"f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91\": rpc error: code = NotFound desc = could not find container \"f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91\": container with ID starting with f29ec0f3b94654a43f8ecd2b72b0ac23828692f35f1b9ef6176608b2a7551d91 not found: ID does not exist" Oct 30 02:41:44.333280 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.333246 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb"] Oct 30 02:41:44.339025 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.338988 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-w4-b93d51-kserve-5c6cc4544f-qqrzb"] Oct 30 02:41:44.376657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.376612 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb"] Oct 30 02:41:44.384606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.384518 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-000edd-6d54b4448f-rb6jb"] Oct 30 02:41:44.416505 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.416450 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw"] Oct 30 02:41:44.420500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.420467 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/premium-w4-b93d51-kserve-65788c994b-jbzmw"] Oct 30 02:41:44.463077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.463034 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4"] Oct 30 02:41:44.466234 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:44.466204 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/distinct-2-w4-b93d51-kserve-55b47d96cd-qvqq4"] Oct 30 02:41:45.654578 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:45.654541 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0590d679-8944-4559-a287-07a90e6d0f42" path="/var/lib/kubelet/pods/0590d679-8944-4559-a287-07a90e6d0f42/volumes" Oct 30 02:41:45.655308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:45.655284 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" path="/var/lib/kubelet/pods/1a1a7bde-671c-4b7f-96d9-a79e994f56c3/volumes" Oct 30 02:41:45.655950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:45.655929 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" path="/var/lib/kubelet/pods/1dab5d1f-f9ea-4b1f-82cf-97ebbd947993/volumes" Oct 30 02:41:45.657002 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:45.656974 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df753c92-98ff-4ceb-8468-6766ded63515" path="/var/lib/kubelet/pods/df753c92-98ff-4ceb-8468-6766ded63515/volumes" Oct 30 02:41:47.629901 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:41:47.629860 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:41:54.061726 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.061687 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm"] Oct 30 02:41:54.062371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062355 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" Oct 30 02:41:54.062371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062370 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062384 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6c9850ac-3004-42da-8ea5-d005e5f5bbca" containerName="payload-pre-processing" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062389 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9850ac-3004-42da-8ea5-d005e5f5bbca" containerName="payload-pre-processing" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062406 5035 cpu_manager.go:414] "containerMap: removing container" podUID="3bf1dda1-3b78-4da2-906b-8882ffee8025" containerName="payload-pre-processing" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062411 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf1dda1-3b78-4da2-906b-8882ffee8025" containerName="payload-pre-processing" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062417 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e98d14cd-45be-457a-a181-3a96931d419e" containerName="maas-api" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062422 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98d14cd-45be-457a-a181-3a96931d419e" containerName="maas-api" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062433 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b6267677-8af0-44d5-9d95-8e8eb629463e" containerName="payload-processing" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062438 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6267677-8af0-44d5-9d95-8e8eb629463e" containerName="payload-processing" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062448 5035 cpu_manager.go:414] "containerMap: removing container" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062454 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062474 5035 cpu_manager.go:414] "containerMap: removing container" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062481 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062495 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062501 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062510 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062515 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062523 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062542 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062558 5035 cpu_manager.go:414] "containerMap: removing container" podUID="59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" containerName="payload-pre-processing" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062565 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" containerName="payload-pre-processing" Oct 30 02:41:54.062567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062582 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062587 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062595 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" containerName="main" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062599 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" containerName="main" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062611 5035 cpu_manager.go:414] "containerMap: removing container" podUID="dca538fb-fd56-45cb-9d60-dc952c70d5ba" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062616 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca538fb-fd56-45cb-9d60-dc952c70d5ba" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062627 5035 cpu_manager.go:414] "containerMap: removing container" podUID="46681de3-fb8d-498e-a12f-2a2eee5f5a40" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062631 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46681de3-fb8d-498e-a12f-2a2eee5f5a40" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062638 5035 cpu_manager.go:414] "containerMap: removing container" podUID="12564f35-a37b-4dd7-befd-91ee03e6b39d" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062643 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12564f35-a37b-4dd7-befd-91ee03e6b39d" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062654 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f1fd086c-87ea-4a37-8ab3-d909e528c75a" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062658 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fd086c-87ea-4a37-8ab3-d909e528c75a" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062668 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e63eff88-dd24-4f72-ac4b-76f163edd204" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062672 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63eff88-dd24-4f72-ac4b-76f163edd204" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062682 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6c0e8297-c571-4105-b69a-c7eb9c3938be" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062687 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0e8297-c571-4105-b69a-c7eb9c3938be" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062696 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062700 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062706 5035 cpu_manager.go:414] "containerMap: removing container" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062711 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062717 5035 cpu_manager.go:414] "containerMap: removing container" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062721 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062730 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062734 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062740 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062745 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062752 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062757 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062763 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d89a119f-d530-4b93-b381-eb23753d22ee" containerName="payload-pre-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062768 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89a119f-d530-4b93-b381-eb23753d22ee" containerName="payload-pre-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062778 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0b3d0792-204a-4487-9ea1-efb6b6616c70" containerName="payload-pre-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062782 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b3d0792-204a-4487-9ea1-efb6b6616c70" containerName="payload-pre-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062790 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062795 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062801 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062806 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062952 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6267677-8af0-44d5-9d95-8e8eb629463e" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062968 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa390600-ff7d-4f05-bc0c-9ada10153b57" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062981 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b60bc0-359c-4074-a4ff-7f561627df13" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.062991 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1c0f8a-ab2a-43dc-bda9-e8461b76facc" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063003 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063014 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0175afca-7f43-471c-9ffb-bd96f1150806" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063025 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c9feb1-42a4-4fa3-8b30-fe3ed5aa69c0" containerName="payload-pre-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063037 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0632efb9-a10a-4f96-8261-2b6ca3e1ffdf" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063048 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1a7bde-671c-4b7f-96d9-a79e994f56c3" containerName="main" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063060 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca538fb-fd56-45cb-9d60-dc952c70d5ba" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063070 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="46681de3-fb8d-498e-a12f-2a2eee5f5a40" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063079 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="12564f35-a37b-4dd7-befd-91ee03e6b39d" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063088 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fd086c-87ea-4a37-8ab3-d909e528c75a" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063096 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63eff88-dd24-4f72-ac4b-76f163edd204" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063106 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0e8297-c571-4105-b69a-c7eb9c3938be" containerName="payload-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063115 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063123 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="4167d08c-361b-4849-bdce-61e7650f13fa" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063131 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="df753c92-98ff-4ceb-8468-6766ded63515" containerName="main" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063140 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063147 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27674de-a2e2-4e4f-82a2-00e44a555270" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063155 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0590d679-8944-4559-a287-07a90e6d0f42" containerName="main" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063162 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89a119f-d530-4b93-b381-eb23753d22ee" containerName="payload-pre-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063169 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b3d0792-204a-4487-9ea1-efb6b6616c70" containerName="payload-pre-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063175 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dab5d1f-f9ea-4b1f-82cf-97ebbd947993" containerName="maas-api" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063181 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9074a6-d3d1-4bd2-901c-f75cbc50b0c3" containerName="istio-proxy" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063187 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9850ac-3004-42da-8ea5-d005e5f5bbca" containerName="payload-pre-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063195 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf1dda1-3b78-4da2-906b-8882ffee8025" containerName="payload-pre-processing" Oct 30 02:41:54.063712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.063203 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98d14cd-45be-457a-a181-3a96931d419e" containerName="maas-api" Oct 30 02:41:54.067163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.067136 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.089058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.089016 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm"] Oct 30 02:41:54.097074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.097035 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-594253\"/\"kube-root-ca.crt\"" Oct 30 02:41:54.097074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.097068 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-594253\"/\"default-dockercfg-hpjkc\"" Oct 30 02:41:54.099194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.099175 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-594253\"/\"rate-test-model-594253-kserve-self-signed-certs\"" Oct 30 02:41:54.099429 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.099413 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-594253\"/\"openshift-service-ca.crt\"" Oct 30 02:41:54.223283 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.223237 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f369a52-a39c-490e-bf37-22bb44483960-tls-certs\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.223461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.223286 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-model-cache\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.223543 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.223479 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-home\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.223591 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.223549 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-tmp-dir\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.223654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.223591 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568lp\" (UniqueName: \"kubernetes.io/projected/2f369a52-a39c-490e-bf37-22bb44483960-kube-api-access-568lp\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.223731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.223712 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-dshm\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.325571 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.325433 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f369a52-a39c-490e-bf37-22bb44483960-tls-certs\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.325571 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.325494 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-model-cache\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.325838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.325622 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-home\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.325838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.325662 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-tmp-dir\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.325838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.325699 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568lp\" (UniqueName: \"kubernetes.io/projected/2f369a52-a39c-490e-bf37-22bb44483960-kube-api-access-568lp\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.325838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.325778 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-dshm\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.326660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.326626 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-home\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.327125 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.327103 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-tmp-dir\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.327395 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.327374 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-model-cache\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.328899 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.328874 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-dshm\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.330258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.330231 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f369a52-a39c-490e-bf37-22bb44483960-tls-certs\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.355007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.354970 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-568lp\" (UniqueName: \"kubernetes.io/projected/2f369a52-a39c-490e-bf37-22bb44483960-kube-api-access-568lp\") pod \"rate-test-model-594253-kserve-57496d79c-4trkm\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.381250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.381206 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:54.703612 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:54.703578 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm"] Oct 30 02:41:54.705217 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:41:54.705191 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f369a52_a39c_490e_bf37_22bb44483960.slice/crio-0d3b31998c2cbd7c05ddf9bda853c4b30e0e06f1d5cd6ece56946bd668a34a38 WatchSource:0}: Error finding container 0d3b31998c2cbd7c05ddf9bda853c4b30e0e06f1d5cd6ece56946bd668a34a38: Status 404 returned error can't find the container with id 0d3b31998c2cbd7c05ddf9bda853c4b30e0e06f1d5cd6ece56946bd668a34a38 Oct 30 02:41:55.365239 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:55.365202 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" event={"ID":"2f369a52-a39c-490e-bf37-22bb44483960","Type":"ContainerStarted","Data":"cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174"} Oct 30 02:41:55.365697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:55.365247 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" event={"ID":"2f369a52-a39c-490e-bf37-22bb44483960","Type":"ContainerStarted","Data":"0d3b31998c2cbd7c05ddf9bda853c4b30e0e06f1d5cd6ece56946bd668a34a38"} Oct 30 02:41:55.381621 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:55.381583 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:41:55.419072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:41:55.419013 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" podStartSLOduration=1.4189944429999999 podStartE2EDuration="1.418994443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:55.416271398 +0000 UTC m=+4393.919203644" watchObservedRunningTime="2026-10-30 02:41:55.418994443 +0000 UTC m=+4393.921926662" Oct 30 02:42:04.381579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:04.381508 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:42:04.395213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:04.395178 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:42:04.443566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:04.443519 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:42:17.631073 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:42:17.631023 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:42:33.430290 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:33.430224 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9"] Oct 30 02:42:38.431380 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.431297 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" podUID="615904ed-4fd5-4af3-9cf3-490f849923df" containerName="payload-pre-processing" containerID="cri-o://be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f" gracePeriod=25 Oct 30 02:42:38.702427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.702397 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:42:38.717814 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.717779 5035 generic.go:356] "Generic (PLEG): container finished" podID="615904ed-4fd5-4af3-9cf3-490f849923df" containerID="be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f" exitCode=0 Oct 30 02:42:38.717998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.717822 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" Oct 30 02:42:38.717998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.717845 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" event={"ID":"615904ed-4fd5-4af3-9cf3-490f849923df","Type":"ContainerDied","Data":"be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f"} Oct 30 02:42:38.717998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.717899 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9" event={"ID":"615904ed-4fd5-4af3-9cf3-490f849923df","Type":"ContainerDied","Data":"90a27a19e9942ec35c6fe34934fc25b953539de7e9363e580bcbdb3dabce9f96"} Oct 30 02:42:38.717998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.717928 5035 scope.go:122] "RemoveContainer" containerID="be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f" Oct 30 02:42:38.730087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.730053 5035 scope.go:122] "RemoveContainer" containerID="be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f" Oct 30 02:42:38.730490 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:42:38.730462 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f\": container with ID starting with be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f not found: ID does not exist" containerID="be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f" Oct 30 02:42:38.730617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.730497 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f"} err="failed to get container status \"be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f\": rpc error: code = NotFound desc = could not find container \"be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f\": container with ID starting with be9f212fd265974a89f671cd5ed46949acef77637c2829a95d8ef6e77a3dac2f not found: ID does not exist" Oct 30 02:42:38.850409 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.850373 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615904ed-4fd5-4af3-9cf3-490f849923df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615904ed-4fd5-4af3-9cf3-490f849923df-plugins-config-volume\") pod \"615904ed-4fd5-4af3-9cf3-490f849923df\" (UID: \"615904ed-4fd5-4af3-9cf3-490f849923df\") " Oct 30 02:42:38.850612 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.850469 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615904ed-4fd5-4af3-9cf3-490f849923df-kube-api-access-stvvx\" (UniqueName: \"kubernetes.io/projected/615904ed-4fd5-4af3-9cf3-490f849923df-kube-api-access-stvvx\") pod \"615904ed-4fd5-4af3-9cf3-490f849923df\" (UID: \"615904ed-4fd5-4af3-9cf3-490f849923df\") " Oct 30 02:42:38.850773 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.850741 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615904ed-4fd5-4af3-9cf3-490f849923df-plugins-config-volume" pod "615904ed-4fd5-4af3-9cf3-490f849923df" (UID: "615904ed-4fd5-4af3-9cf3-490f849923df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:38.850899 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.850862 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615904ed-4fd5-4af3-9cf3-490f849923df-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:42:38.852706 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.852683 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615904ed-4fd5-4af3-9cf3-490f849923df-kube-api-access-stvvx" pod "615904ed-4fd5-4af3-9cf3-490f849923df" (UID: "615904ed-4fd5-4af3-9cf3-490f849923df"). InnerVolumeSpecName "kube-api-access-stvvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:38.952599 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:38.952470 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvvx\" (UniqueName: \"kubernetes.io/projected/615904ed-4fd5-4af3-9cf3-490f849923df-kube-api-access-stvvx\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:42:39.066540 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:39.066480 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9"] Oct 30 02:42:39.078341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:39.078301 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a0684f-84d555f9d7-25wf9"] Oct 30 02:42:39.653538 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:39.653499 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615904ed-4fd5-4af3-9cf3-490f849923df" path="/var/lib/kubelet/pods/615904ed-4fd5-4af3-9cf3-490f849923df/volumes" Oct 30 02:42:47.640873 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:42:47.640772 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:42:58.454136 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.453914 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9"] Oct 30 02:42:58.455123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.454702 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" podUID="8e72d15b-0f2c-4bbe-9dea-add735e9b124" containerName="maas-api" containerID="cri-o://1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba" gracePeriod=30 Oct 30 02:42:58.769520 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.769483 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:42:58.848996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.848941 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-maas-api-tls\") pod \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " Oct 30 02:42:58.850171 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.849591 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e72d15b-0f2c-4bbe-9dea-add735e9b124-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e72d15b-0f2c-4bbe-9dea-add735e9b124-openshift-service-ca\") pod \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " Oct 30 02:42:58.850570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.850109 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e72d15b-0f2c-4bbe-9dea-add735e9b124-openshift-service-ca" pod "8e72d15b-0f2c-4bbe-9dea-add735e9b124" (UID: "8e72d15b-0f2c-4bbe-9dea-add735e9b124"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:58.857223 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.856483 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e72d15b-0f2c-4bbe-9dea-add735e9b124-kube-api-access-j892t\" (UniqueName: \"kubernetes.io/projected/8e72d15b-0f2c-4bbe-9dea-add735e9b124-kube-api-access-j892t\") pod \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " Oct 30 02:42:58.857223 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.856566 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-metrics-certs\") pod \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\" (UID: \"8e72d15b-0f2c-4bbe-9dea-add735e9b124\") " Oct 30 02:42:58.857463 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.857373 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e72d15b-0f2c-4bbe-9dea-add735e9b124-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:42:58.864354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.864261 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-maas-api-tls" pod "8e72d15b-0f2c-4bbe-9dea-add735e9b124" (UID: "8e72d15b-0f2c-4bbe-9dea-add735e9b124"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:58.868921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.868884 5035 generic.go:356] "Generic (PLEG): container finished" podID="8e72d15b-0f2c-4bbe-9dea-add735e9b124" containerID="1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba" exitCode=0 Oct 30 02:42:58.869112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.868943 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" Oct 30 02:42:58.869112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.869060 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" event={"ID":"8e72d15b-0f2c-4bbe-9dea-add735e9b124","Type":"ContainerDied","Data":"1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba"} Oct 30 02:42:58.869245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.869110 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9" event={"ID":"8e72d15b-0f2c-4bbe-9dea-add735e9b124","Type":"ContainerDied","Data":"1c94a3ed31a28e13a00335ee264743d5245c7df9b77c923050be48fb0cfd7669"} Oct 30 02:42:58.869245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.869142 5035 scope.go:122] "RemoveContainer" containerID="1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba" Oct 30 02:42:58.876687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.876258 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-metrics-certs" pod "8e72d15b-0f2c-4bbe-9dea-add735e9b124" (UID: "8e72d15b-0f2c-4bbe-9dea-add735e9b124"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:58.877519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.877468 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e72d15b-0f2c-4bbe-9dea-add735e9b124-kube-api-access-j892t" pod "8e72d15b-0f2c-4bbe-9dea-add735e9b124" (UID: "8e72d15b-0f2c-4bbe-9dea-add735e9b124"). InnerVolumeSpecName "kube-api-access-j892t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:58.898627 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.898595 5035 scope.go:122] "RemoveContainer" containerID="1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba" Oct 30 02:42:58.899053 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:42:58.899033 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba\": container with ID starting with 1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba not found: ID does not exist" containerID="1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba" Oct 30 02:42:58.899119 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.899062 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba"} err="failed to get container status \"1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba\": rpc error: code = NotFound desc = could not find container \"1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba\": container with ID starting with 1078d05bdd66305d6d71655d3539dc99557ea875de2b1143330d5ec629a31aba not found: ID does not exist" Oct 30 02:42:58.958960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.958917 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j892t\" (UniqueName: \"kubernetes.io/projected/8e72d15b-0f2c-4bbe-9dea-add735e9b124-kube-api-access-j892t\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:42:58.958960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.958955 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:42:58.958960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:58.958965 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e72d15b-0f2c-4bbe-9dea-add735e9b124-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:42:59.225134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:59.225091 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9"] Oct 30 02:42:59.232421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:59.232373 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a0684f-5c6cf7dd94-6bkq9"] Oct 30 02:42:59.665263 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:42:59.665222 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e72d15b-0f2c-4bbe-9dea-add735e9b124" path="/var/lib/kubelet/pods/8e72d15b-0f2c-4bbe-9dea-add735e9b124/volumes" Oct 30 02:43:14.816299 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:14.816256 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z"] Oct 30 02:43:14.853699 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:14.853656 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz"] Oct 30 02:43:17.634052 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:43:17.634014 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:43:19.817385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:19.817189 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" podUID="957eece0-9249-4035-ad6b-a8e2a967e3a8" containerName="payload-processing" containerID="cri-o://1a99f001f01854d01f8f91485020ce691c183871745b8a4e54bdd6dd165b0a79" gracePeriod=25 Oct 30 02:43:19.854783 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:19.854724 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" podUID="e2d34e97-6a00-4733-8296-f16c73833283" containerName="payload-pre-processing" containerID="cri-o://a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317" gracePeriod=25 Oct 30 02:43:20.047572 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:20.044914 5035 generic.go:356] "Generic (PLEG): container finished" podID="957eece0-9249-4035-ad6b-a8e2a967e3a8" containerID="1a99f001f01854d01f8f91485020ce691c183871745b8a4e54bdd6dd165b0a79" exitCode=0 Oct 30 02:43:20.047572 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:20.045226 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" event={"ID":"957eece0-9249-4035-ad6b-a8e2a967e3a8","Type":"ContainerDied","Data":"1a99f001f01854d01f8f91485020ce691c183871745b8a4e54bdd6dd165b0a79"} Oct 30 02:43:20.109353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:20.109323 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 02:43:20.263791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:20.263755 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957eece0-9249-4035-ad6b-a8e2a967e3a8-kube-api-access-l4g9p\" (UniqueName: \"kubernetes.io/projected/957eece0-9249-4035-ad6b-a8e2a967e3a8-kube-api-access-l4g9p\") pod \"957eece0-9249-4035-ad6b-a8e2a967e3a8\" (UID: \"957eece0-9249-4035-ad6b-a8e2a967e3a8\") " Oct 30 02:43:20.263996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:20.263852 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957eece0-9249-4035-ad6b-a8e2a967e3a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957eece0-9249-4035-ad6b-a8e2a967e3a8-plugins-config-volume\") pod \"957eece0-9249-4035-ad6b-a8e2a967e3a8\" (UID: \"957eece0-9249-4035-ad6b-a8e2a967e3a8\") " Oct 30 02:43:20.264279 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:20.264253 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957eece0-9249-4035-ad6b-a8e2a967e3a8-plugins-config-volume" pod "957eece0-9249-4035-ad6b-a8e2a967e3a8" (UID: "957eece0-9249-4035-ad6b-a8e2a967e3a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:43:20.266098 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:20.266075 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957eece0-9249-4035-ad6b-a8e2a967e3a8-kube-api-access-l4g9p" pod "957eece0-9249-4035-ad6b-a8e2a967e3a8" (UID: "957eece0-9249-4035-ad6b-a8e2a967e3a8"). InnerVolumeSpecName "kube-api-access-l4g9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:20.365750 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:20.365658 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4g9p\" (UniqueName: \"kubernetes.io/projected/957eece0-9249-4035-ad6b-a8e2a967e3a8-kube-api-access-l4g9p\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:43:20.365750 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:20.365691 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957eece0-9249-4035-ad6b-a8e2a967e3a8-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:43:21.053994 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:21.053958 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" Oct 30 02:43:21.054482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:21.053987 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z" event={"ID":"957eece0-9249-4035-ad6b-a8e2a967e3a8","Type":"ContainerDied","Data":"1a303971e6f538f3b0a963a751ba2828204c7c6fd75e53285fa0bef579b4f8b2"} Oct 30 02:43:21.054482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:21.054055 5035 scope.go:122] "RemoveContainer" containerID="1a99f001f01854d01f8f91485020ce691c183871745b8a4e54bdd6dd165b0a79" Oct 30 02:43:21.108053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:21.108009 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z"] Oct 30 02:43:21.119730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:21.119683 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f6f4f0-7447ff75b-wxn5z"] Oct 30 02:43:21.644208 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:21.644174 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957eece0-9249-4035-ad6b-a8e2a967e3a8" path="/var/lib/kubelet/pods/957eece0-9249-4035-ad6b-a8e2a967e3a8/volumes" Oct 30 02:43:29.027341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:29.027294 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" podUID="e2d34e97-6a00-4733-8296-f16c73833283" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:43:39.027262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:39.027214 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" podUID="e2d34e97-6a00-4733-8296-f16c73833283" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:43:40.137834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.137805 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 02:43:40.197581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.197513 5035 generic.go:356] "Generic (PLEG): container finished" podID="e2d34e97-6a00-4733-8296-f16c73833283" containerID="a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317" exitCode=0 Oct 30 02:43:40.197581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.197573 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" event={"ID":"e2d34e97-6a00-4733-8296-f16c73833283","Type":"ContainerDied","Data":"a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317"} Oct 30 02:43:40.197889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.197610 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" Oct 30 02:43:40.197889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.197637 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz" event={"ID":"e2d34e97-6a00-4733-8296-f16c73833283","Type":"ContainerDied","Data":"d07e2e029ea6c3969136d9c4d36c7f0de6d7ec926864770601b3b52b9d680b85"} Oct 30 02:43:40.197889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.197668 5035 scope.go:122] "RemoveContainer" containerID="a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317" Oct 30 02:43:40.214335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.214304 5035 scope.go:122] "RemoveContainer" containerID="a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317" Oct 30 02:43:40.214768 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:43:40.214738 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317\": container with ID starting with a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317 not found: ID does not exist" containerID="a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317" Oct 30 02:43:40.214869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.214778 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317"} err="failed to get container status \"a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317\": rpc error: code = NotFound desc = could not find container \"a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317\": container with ID starting with a67cb528417bc4c854bbfdb49e82828fd331dda5032fc8c818ac90d129e80317 not found: ID does not exist" Oct 30 02:43:40.234736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.234691 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d34e97-6a00-4733-8296-f16c73833283-kube-api-access-fkmfz\" (UniqueName: \"kubernetes.io/projected/e2d34e97-6a00-4733-8296-f16c73833283-kube-api-access-fkmfz\") pod \"e2d34e97-6a00-4733-8296-f16c73833283\" (UID: \"e2d34e97-6a00-4733-8296-f16c73833283\") " Oct 30 02:43:40.234935 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.234754 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d34e97-6a00-4733-8296-f16c73833283-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d34e97-6a00-4733-8296-f16c73833283-plugins-config-volume\") pod \"e2d34e97-6a00-4733-8296-f16c73833283\" (UID: \"e2d34e97-6a00-4733-8296-f16c73833283\") " Oct 30 02:43:40.235213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.235181 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d34e97-6a00-4733-8296-f16c73833283-plugins-config-volume" pod "e2d34e97-6a00-4733-8296-f16c73833283" (UID: "e2d34e97-6a00-4733-8296-f16c73833283"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:43:40.235387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.235369 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d34e97-6a00-4733-8296-f16c73833283-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:43:40.237139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.237105 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d34e97-6a00-4733-8296-f16c73833283-kube-api-access-fkmfz" pod "e2d34e97-6a00-4733-8296-f16c73833283" (UID: "e2d34e97-6a00-4733-8296-f16c73833283"). InnerVolumeSpecName "kube-api-access-fkmfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:40.336938 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.336896 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkmfz\" (UniqueName: \"kubernetes.io/projected/e2d34e97-6a00-4733-8296-f16c73833283-kube-api-access-fkmfz\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:43:40.544347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.544300 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz"] Oct 30 02:43:40.556240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:40.556200 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f6f4f0-6bfcfffbc9-b6cdz"] Oct 30 02:43:41.646018 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:41.645985 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d34e97-6a00-4733-8296-f16c73833283" path="/var/lib/kubelet/pods/e2d34e97-6a00-4733-8296-f16c73833283/volumes" Oct 30 02:43:44.112462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:44.112434 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:43:44.123293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:44.123268 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:43:44.128072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:44.128045 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:43:44.181984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:44.181958 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:43:44.193396 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:44.193368 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:43:44.198290 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:44.198262 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:43:47.639273 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:43:47.639231 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:43:47.649625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:47.649589 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2"] Oct 30 02:43:47.649853 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:47.649827 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" podUID="e4cae86e-0335-4f80-8521-fec459fe858f" containerName="maas-api" containerID="cri-o://184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90" gracePeriod=30 Oct 30 02:43:47.959797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:47.959763 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 02:43:48.052048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.051980 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-metrics-certs\") pod \"e4cae86e-0335-4f80-8521-fec459fe858f\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " Oct 30 02:43:48.052406 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.052382 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cae86e-0335-4f80-8521-fec459fe858f-kube-api-access-jclcx\" (UniqueName: \"kubernetes.io/projected/e4cae86e-0335-4f80-8521-fec459fe858f-kube-api-access-jclcx\") pod \"e4cae86e-0335-4f80-8521-fec459fe858f\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " Oct 30 02:43:48.052823 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.052788 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cae86e-0335-4f80-8521-fec459fe858f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cae86e-0335-4f80-8521-fec459fe858f-openshift-service-ca\") pod \"e4cae86e-0335-4f80-8521-fec459fe858f\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " Oct 30 02:43:48.053228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.053194 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls\") pod \"e4cae86e-0335-4f80-8521-fec459fe858f\" (UID: \"e4cae86e-0335-4f80-8521-fec459fe858f\") " Oct 30 02:43:48.056456 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.056424 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cae86e-0335-4f80-8521-fec459fe858f-openshift-service-ca" pod "e4cae86e-0335-4f80-8521-fec459fe858f" (UID: "e4cae86e-0335-4f80-8521-fec459fe858f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:43:48.061235 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.061182 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls" pod "e4cae86e-0335-4f80-8521-fec459fe858f" (UID: "e4cae86e-0335-4f80-8521-fec459fe858f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:43:48.061627 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.061584 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cae86e-0335-4f80-8521-fec459fe858f-kube-api-access-jclcx" pod "e4cae86e-0335-4f80-8521-fec459fe858f" (UID: "e4cae86e-0335-4f80-8521-fec459fe858f"). InnerVolumeSpecName "kube-api-access-jclcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:48.075735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.075674 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-metrics-certs" pod "e4cae86e-0335-4f80-8521-fec459fe858f" (UID: "e4cae86e-0335-4f80-8521-fec459fe858f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:43:48.156626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.156516 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cae86e-0335-4f80-8521-fec459fe858f-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:43:48.156626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.156629 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:43:48.156866 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.156641 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cae86e-0335-4f80-8521-fec459fe858f-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:43:48.156866 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.156650 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jclcx\" (UniqueName: \"kubernetes.io/projected/e4cae86e-0335-4f80-8521-fec459fe858f-kube-api-access-jclcx\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:43:48.267790 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.267747 5035 generic.go:356] "Generic (PLEG): container finished" podID="e4cae86e-0335-4f80-8521-fec459fe858f" containerID="184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90" exitCode=0 Oct 30 02:43:48.268005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.267816 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" Oct 30 02:43:48.268005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.267829 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" event={"ID":"e4cae86e-0335-4f80-8521-fec459fe858f","Type":"ContainerDied","Data":"184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90"} Oct 30 02:43:48.268005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.267920 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2" event={"ID":"e4cae86e-0335-4f80-8521-fec459fe858f","Type":"ContainerDied","Data":"3e7f98f13c06ac3d5d47597d61d43669e7adeaa39954c8c51da511e21cc0eba7"} Oct 30 02:43:48.268005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.267954 5035 scope.go:122] "RemoveContainer" containerID="184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90" Oct 30 02:43:48.279478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.279454 5035 scope.go:122] "RemoveContainer" containerID="184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90" Oct 30 02:43:48.279838 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:43:48.279814 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90\": container with ID starting with 184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90 not found: ID does not exist" containerID="184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90" Oct 30 02:43:48.279897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.279849 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90"} err="failed to get container status \"184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90\": rpc error: code = NotFound desc = could not find container \"184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90\": container with ID starting with 184bdaff1d79a1274288e64f3c45d57d94f2f3fd8028ca809d2a0b72760b3c90 not found: ID does not exist" Oct 30 02:43:48.320362 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.320319 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2"] Oct 30 02:43:48.326641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:48.326603 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f6f4f0-68c766f55f-gxxs2"] Oct 30 02:43:49.644095 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:43:49.644060 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cae86e-0335-4f80-8521-fec459fe858f" path="/var/lib/kubelet/pods/e4cae86e-0335-4f80-8521-fec459fe858f/volumes" Oct 30 02:44:17.630187 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:44:17.630147 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:44:44.625897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:44.625269 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l"] Oct 30 02:44:44.625897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:44.625626 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" podUID="85f6104d-2e09-41f1-8bc8-6e2fe8c28160" containerName="discovery" containerID="cri-o://37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4" gracePeriod=30 Oct 30 02:44:44.686427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:44.686392 5035 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xr74l container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:8080/ready\": dial tcp 10.132.0.102:8080: connect: connection refused" start-of-body= Oct 30 02:44:44.686639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:44.686465 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" podUID="85f6104d-2e09-41f1-8bc8-6e2fe8c28160" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.102:8080/ready\": dial tcp 10.132.0.102:8080: connect: connection refused" Oct 30 02:44:44.927053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:44.927014 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 02:44:45.048085 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.048048 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-token\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-token\") pod \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " Oct 30 02:44:45.048271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.048113 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-dns-cert\") pod \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " Oct 30 02:44:45.048271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.048137 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-kube-api-access-rkc9m\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-kube-api-access-rkc9m\") pod \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " Oct 30 02:44:45.048271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.048233 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-kubeconfig\") pod \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " Oct 30 02:44:45.048381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.048340 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-ca-configmap\") pod \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " Oct 30 02:44:45.048482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.048457 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-local-certs\") pod \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " Oct 30 02:44:45.048586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.048572 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-cacerts\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-cacerts\") pod \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\" (UID: \"85f6104d-2e09-41f1-8bc8-6e2fe8c28160\") " Oct 30 02:44:45.048761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.048735 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-ca-configmap" pod "85f6104d-2e09-41f1-8bc8-6e2fe8c28160" (UID: "85f6104d-2e09-41f1-8bc8-6e2fe8c28160"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:44:45.049248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.049222 5035 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-ca-configmap\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:44:45.050829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.050784 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-kube-api-access-rkc9m" pod "85f6104d-2e09-41f1-8bc8-6e2fe8c28160" (UID: "85f6104d-2e09-41f1-8bc8-6e2fe8c28160"). InnerVolumeSpecName "kube-api-access-rkc9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:44:45.050941 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.050901 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-dns-cert" pod "85f6104d-2e09-41f1-8bc8-6e2fe8c28160" (UID: "85f6104d-2e09-41f1-8bc8-6e2fe8c28160"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:44:45.051065 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.051039 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-token" pod "85f6104d-2e09-41f1-8bc8-6e2fe8c28160" (UID: "85f6104d-2e09-41f1-8bc8-6e2fe8c28160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:44:45.051180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.051088 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-local-certs" pod "85f6104d-2e09-41f1-8bc8-6e2fe8c28160" (UID: "85f6104d-2e09-41f1-8bc8-6e2fe8c28160"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:45.051180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.051158 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-kubeconfig" pod "85f6104d-2e09-41f1-8bc8-6e2fe8c28160" (UID: "85f6104d-2e09-41f1-8bc8-6e2fe8c28160"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:44:45.051304 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.051195 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-cacerts" pod "85f6104d-2e09-41f1-8bc8-6e2fe8c28160" (UID: "85f6104d-2e09-41f1-8bc8-6e2fe8c28160"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:44:45.150191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.150097 5035 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-kubeconfig\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:44:45.150191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.150135 5035 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-local-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:44:45.150191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.150146 5035 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-cacerts\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:44:45.150191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.150153 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:44:45.150191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.150162 5035 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-istio-csr-dns-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:44:45.150191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.150174 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkc9m\" (UniqueName: \"kubernetes.io/projected/85f6104d-2e09-41f1-8bc8-6e2fe8c28160-kube-api-access-rkc9m\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:44:45.687854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.687814 5035 generic.go:356] "Generic (PLEG): container finished" podID="85f6104d-2e09-41f1-8bc8-6e2fe8c28160" containerID="37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4" exitCode=0 Oct 30 02:44:45.688353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.687862 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" Oct 30 02:44:45.688353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.687904 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" event={"ID":"85f6104d-2e09-41f1-8bc8-6e2fe8c28160","Type":"ContainerDied","Data":"37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4"} Oct 30 02:44:45.688353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.687973 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l" event={"ID":"85f6104d-2e09-41f1-8bc8-6e2fe8c28160","Type":"ContainerDied","Data":"e1195e2d45ffbf510ffcf361e29ea77e608574578fe1b2fa39afbed52abde40f"} Oct 30 02:44:45.688353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.688004 5035 scope.go:122] "RemoveContainer" containerID="37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4" Oct 30 02:44:45.706703 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.706673 5035 scope.go:122] "RemoveContainer" containerID="37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4" Oct 30 02:44:45.707047 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:44:45.707025 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4\": container with ID starting with 37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4 not found: ID does not exist" containerID="37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4" Oct 30 02:44:45.707130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.707060 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4"} err="failed to get container status \"37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4\": rpc error: code = NotFound desc = could not find container \"37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4\": container with ID starting with 37d0c75dfbb681497056c28a1dd6c6fb730d82d172b7279ba70e5ace563a11a4 not found: ID does not exist" Oct 30 02:44:45.740635 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.740594 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l"] Oct 30 02:44:45.752611 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:45.752573 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr74l"] Oct 30 02:44:47.630655 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:44:47.630610 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:44:47.648119 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:47.648079 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f6104d-2e09-41f1-8bc8-6e2fe8c28160" path="/var/lib/kubelet/pods/85f6104d-2e09-41f1-8bc8-6e2fe8c28160/volumes" Oct 30 02:44:50.690226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690182 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11184-01ba9d"] Oct 30 02:44:50.690868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690850 5035 cpu_manager.go:414] "containerMap: removing container" podUID="85f6104d-2e09-41f1-8bc8-6e2fe8c28160" containerName="discovery" Oct 30 02:44:50.690913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690869 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f6104d-2e09-41f1-8bc8-6e2fe8c28160" containerName="discovery" Oct 30 02:44:50.690913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690882 5035 cpu_manager.go:414] "containerMap: removing container" podUID="615904ed-4fd5-4af3-9cf3-490f849923df" containerName="payload-pre-processing" Oct 30 02:44:50.690913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690887 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615904ed-4fd5-4af3-9cf3-490f849923df" containerName="payload-pre-processing" Oct 30 02:44:50.690913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690899 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e2d34e97-6a00-4733-8296-f16c73833283" containerName="payload-pre-processing" Oct 30 02:44:50.690913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690904 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d34e97-6a00-4733-8296-f16c73833283" containerName="payload-pre-processing" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690937 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e4cae86e-0335-4f80-8521-fec459fe858f" containerName="maas-api" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690942 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cae86e-0335-4f80-8521-fec459fe858f" containerName="maas-api" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690954 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8e72d15b-0f2c-4bbe-9dea-add735e9b124" containerName="maas-api" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690959 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e72d15b-0f2c-4bbe-9dea-add735e9b124" containerName="maas-api" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690974 5035 cpu_manager.go:414] "containerMap: removing container" podUID="957eece0-9249-4035-ad6b-a8e2a967e3a8" containerName="payload-processing" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.690978 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957eece0-9249-4035-ad6b-a8e2a967e3a8" containerName="payload-processing" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.691065 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a7ded3-59ff-4e1c-bb3a-00956659d646" containerName="istio-proxy" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.691072 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cae86e-0335-4f80-8521-fec459fe858f" containerName="maas-api" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.691080 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e72d15b-0f2c-4bbe-9dea-add735e9b124" containerName="maas-api" Oct 30 02:44:50.691093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.691089 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="957eece0-9249-4035-ad6b-a8e2a967e3a8" containerName="payload-processing" Oct 30 02:44:50.691408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.691098 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f6104d-2e09-41f1-8bc8-6e2fe8c28160" containerName="discovery" Oct 30 02:44:50.691408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.691106 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="615904ed-4fd5-4af3-9cf3-490f849923df" containerName="payload-pre-processing" Oct 30 02:44:50.691408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.691111 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d34e97-6a00-4733-8296-f16c73833283" containerName="payload-pre-processing" Oct 30 02:44:50.695866 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.695844 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-01ba9d" Oct 30 02:44:50.713896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.713859 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 02:44:50.714156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.714128 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11184-01ba9d"] Oct 30 02:44:50.722395 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.722365 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 02:44:50.723123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.723093 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9wh\" (UniqueName: \"kubernetes.io/projected/49e7a56b-3fe8-41ba-98d7-eb3578de11dd-kube-api-access-4v9wh\") pod \"test-curl-11184-01ba9d\" (UID: \"49e7a56b-3fe8-41ba-98d7-eb3578de11dd\") " pod="opendatahub/test-curl-11184-01ba9d" Oct 30 02:44:50.725824 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.725802 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sddb\"" Oct 30 02:44:50.824704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.824661 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9wh\" (UniqueName: \"kubernetes.io/projected/49e7a56b-3fe8-41ba-98d7-eb3578de11dd-kube-api-access-4v9wh\") pod \"test-curl-11184-01ba9d\" (UID: \"49e7a56b-3fe8-41ba-98d7-eb3578de11dd\") " pod="opendatahub/test-curl-11184-01ba9d" Oct 30 02:44:50.848039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:50.848001 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9wh\" (UniqueName: \"kubernetes.io/projected/49e7a56b-3fe8-41ba-98d7-eb3578de11dd-kube-api-access-4v9wh\") pod \"test-curl-11184-01ba9d\" (UID: \"49e7a56b-3fe8-41ba-98d7-eb3578de11dd\") " pod="opendatahub/test-curl-11184-01ba9d" Oct 30 02:44:51.011356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:51.011306 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-01ba9d" Oct 30 02:44:51.320508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:51.320472 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11184-01ba9d"] Oct 30 02:44:51.322251 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:44:51.322218 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49e7a56b_3fe8_41ba_98d7_eb3578de11dd.slice/crio-dfe20f25002654b95dcca829a35fc5b1773b52acf8bc1078902192d164b905e5 WatchSource:0}: Error finding container dfe20f25002654b95dcca829a35fc5b1773b52acf8bc1078902192d164b905e5: Status 404 returned error can't find the container with id dfe20f25002654b95dcca829a35fc5b1773b52acf8bc1078902192d164b905e5 Oct 30 02:44:51.324341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:51.324321 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:44:51.735896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:51.735853 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11184-01ba9d" event={"ID":"49e7a56b-3fe8-41ba-98d7-eb3578de11dd","Type":"ContainerStarted","Data":"dfe20f25002654b95dcca829a35fc5b1773b52acf8bc1078902192d164b905e5"} Oct 30 02:44:53.754177 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:53.754136 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11184-01ba9d" event={"ID":"49e7a56b-3fe8-41ba-98d7-eb3578de11dd","Type":"ContainerStarted","Data":"d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656"} Oct 30 02:44:53.792794 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:53.792721 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11184-01ba9d" podStartSLOduration=1.7099906219999998 podStartE2EDuration="3.792704154s" totalImagesPullingTime="2.082713532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:44:53.791471531 +0000 UTC m=+4572.294403743" watchObservedRunningTime="2026-10-30 02:44:53.792704154 +0000 UTC m=+4572.295636374" Oct 30 02:44:54.632456 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.632412 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11184-01ba9d"] Oct 30 02:44:54.639474 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.639435 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11184-01ba9d"] Oct 30 02:44:54.787832 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.787787 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11184-102e4f"] Oct 30 02:44:54.788704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.788678 5035 cpu_manager.go:414] "containerMap: removing container" podUID="49e7a56b-3fe8-41ba-98d7-eb3578de11dd" containerName="test-curl-11184-01ba9d" Oct 30 02:44:54.788852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.788704 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e7a56b-3fe8-41ba-98d7-eb3578de11dd" containerName="test-curl-11184-01ba9d" Oct 30 02:44:54.788950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.788935 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e7a56b-3fe8-41ba-98d7-eb3578de11dd" containerName="test-curl-11184-01ba9d" Oct 30 02:44:54.792780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.792752 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-102e4f" Oct 30 02:44:54.812587 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.810149 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11184-102e4f"] Oct 30 02:44:54.881096 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.881044 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp9qz\" (UniqueName: \"kubernetes.io/projected/b87fcf2c-3bf3-43be-a5d7-b36ffba88f31-kube-api-access-vp9qz\") pod \"test-curl-11184-102e4f\" (UID: \"b87fcf2c-3bf3-43be-a5d7-b36ffba88f31\") " pod="opendatahub/test-curl-11184-102e4f" Oct 30 02:44:54.982719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:54.982684 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9qz\" (UniqueName: \"kubernetes.io/projected/b87fcf2c-3bf3-43be-a5d7-b36ffba88f31-kube-api-access-vp9qz\") pod \"test-curl-11184-102e4f\" (UID: \"b87fcf2c-3bf3-43be-a5d7-b36ffba88f31\") " pod="opendatahub/test-curl-11184-102e4f" Oct 30 02:44:55.005981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:55.005948 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9qz\" (UniqueName: \"kubernetes.io/projected/b87fcf2c-3bf3-43be-a5d7-b36ffba88f31-kube-api-access-vp9qz\") pod \"test-curl-11184-102e4f\" (UID: \"b87fcf2c-3bf3-43be-a5d7-b36ffba88f31\") " pod="opendatahub/test-curl-11184-102e4f" Oct 30 02:44:55.110112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:55.109913 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-102e4f" Oct 30 02:44:55.440770 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:55.440671 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11184-102e4f"] Oct 30 02:44:55.440940 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:44:55.440767 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb87fcf2c_3bf3_43be_a5d7_b36ffba88f31.slice/crio-c51a00b23448993e039045e0d0f0473eede879f5c7fc8ee79c86fe991f282aa8 WatchSource:0}: Error finding container c51a00b23448993e039045e0d0f0473eede879f5c7fc8ee79c86fe991f282aa8: Status 404 returned error can't find the container with id c51a00b23448993e039045e0d0f0473eede879f5c7fc8ee79c86fe991f282aa8 Oct 30 02:44:55.769258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:55.769207 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11184-102e4f" event={"ID":"b87fcf2c-3bf3-43be-a5d7-b36ffba88f31","Type":"ContainerStarted","Data":"cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342"} Oct 30 02:44:55.769258 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:55.769264 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11184-102e4f" event={"ID":"b87fcf2c-3bf3-43be-a5d7-b36ffba88f31","Type":"ContainerStarted","Data":"c51a00b23448993e039045e0d0f0473eede879f5c7fc8ee79c86fe991f282aa8"} Oct 30 02:44:55.769582 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:55.769454 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11184-01ba9d" podUID="49e7a56b-3fe8-41ba-98d7-eb3578de11dd" containerName="test-curl-11184-01ba9d" containerID="cri-o://d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656" gracePeriod=2 Oct 30 02:44:55.809965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:55.809899 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11184-102e4f" podStartSLOduration=1.809883098 podStartE2EDuration="1.809883098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:44:55.806506936 +0000 UTC m=+4574.309439176" watchObservedRunningTime="2026-10-30 02:44:55.809883098 +0000 UTC m=+4574.312815318" Oct 30 02:44:56.609912 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:56.609868 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11184-102e4f"] Oct 30 02:44:56.615063 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:56.615019 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11184-102e4f"] Oct 30 02:44:57.783563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:57.783509 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11184-102e4f" podUID="b87fcf2c-3bf3-43be-a5d7-b36ffba88f31" containerName="test-curl-11184-102e4f" containerID="cri-o://cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342" gracePeriod=2 Oct 30 02:44:58.019472 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:58.019440 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-01ba9d" Oct 30 02:44:58.123745 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:58.123645 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e7a56b-3fe8-41ba-98d7-eb3578de11dd-kube-api-access-4v9wh\" (UniqueName: \"kubernetes.io/projected/49e7a56b-3fe8-41ba-98d7-eb3578de11dd-kube-api-access-4v9wh\") pod \"49e7a56b-3fe8-41ba-98d7-eb3578de11dd\" (UID: \"49e7a56b-3fe8-41ba-98d7-eb3578de11dd\") " Oct 30 02:44:58.125977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:58.125941 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e7a56b-3fe8-41ba-98d7-eb3578de11dd-kube-api-access-4v9wh" pod "49e7a56b-3fe8-41ba-98d7-eb3578de11dd" (UID: "49e7a56b-3fe8-41ba-98d7-eb3578de11dd"). InnerVolumeSpecName "kube-api-access-4v9wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:44:58.232550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:58.226029 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v9wh\" (UniqueName: \"kubernetes.io/projected/49e7a56b-3fe8-41ba-98d7-eb3578de11dd-kube-api-access-4v9wh\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:44:58.791991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:58.791946 5035 generic.go:356] "Generic (PLEG): container finished" podID="49e7a56b-3fe8-41ba-98d7-eb3578de11dd" containerID="d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656" exitCode=137 Oct 30 02:44:58.792481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:58.792018 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-01ba9d" Oct 30 02:44:58.792481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:58.792046 5035 scope.go:122] "RemoveContainer" containerID="d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656" Oct 30 02:44:58.803037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:58.803010 5035 scope.go:122] "RemoveContainer" containerID="d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656" Oct 30 02:44:58.803402 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:44:58.803378 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656\": container with ID starting with d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656 not found: ID does not exist" containerID="d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656" Oct 30 02:44:58.803464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:58.803412 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656"} err="failed to get container status \"d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656\": rpc error: code = NotFound desc = could not find container \"d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656\": container with ID starting with d2404987024b2f605f315fe66f8929948369e23cf30cf8835e27d4a861912656 not found: ID does not exist" Oct 30 02:44:58.814064 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:44:58.814021 5035 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11184-01ba9d\" is forbidden: User \"system:node:ip-10-0-142-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-110.ec2.internal' and this object" podUID="49e7a56b-3fe8-41ba-98d7-eb3578de11dd" pod="opendatahub/test-curl-11184-01ba9d" Oct 30 02:44:59.644175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:44:59.644137 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e7a56b-3fe8-41ba-98d7-eb3578de11dd" path="/var/lib/kubelet/pods/49e7a56b-3fe8-41ba-98d7-eb3578de11dd/volumes" Oct 30 02:45:00.007584 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.007495 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-102e4f" Oct 30 02:45:00.149977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.149937 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87fcf2c-3bf3-43be-a5d7-b36ffba88f31-kube-api-access-vp9qz\" (UniqueName: \"kubernetes.io/projected/b87fcf2c-3bf3-43be-a5d7-b36ffba88f31-kube-api-access-vp9qz\") pod \"b87fcf2c-3bf3-43be-a5d7-b36ffba88f31\" (UID: \"b87fcf2c-3bf3-43be-a5d7-b36ffba88f31\") " Oct 30 02:45:00.152160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.152123 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87fcf2c-3bf3-43be-a5d7-b36ffba88f31-kube-api-access-vp9qz" pod "b87fcf2c-3bf3-43be-a5d7-b36ffba88f31" (UID: "b87fcf2c-3bf3-43be-a5d7-b36ffba88f31"). InnerVolumeSpecName "kube-api-access-vp9qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:00.220548 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.220101 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf"] Oct 30 02:45:00.221045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.221015 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b87fcf2c-3bf3-43be-a5d7-b36ffba88f31" containerName="test-curl-11184-102e4f" Oct 30 02:45:00.221045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.221045 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87fcf2c-3bf3-43be-a5d7-b36ffba88f31" containerName="test-curl-11184-102e4f" Oct 30 02:45:00.221276 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.221198 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87fcf2c-3bf3-43be-a5d7-b36ffba88f31" containerName="test-curl-11184-102e4f" Oct 30 02:45:00.225420 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.225395 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:00.251873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.251827 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp9qz\" (UniqueName: \"kubernetes.io/projected/b87fcf2c-3bf3-43be-a5d7-b36ffba88f31-kube-api-access-vp9qz\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:00.258667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.258637 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:45:00.273889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.273840 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq"] Oct 30 02:45:00.284971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.284903 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf"] Oct 30 02:45:00.285717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.285686 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:00.342714 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.342666 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq"] Oct 30 02:45:00.347512 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.347471 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g"] Oct 30 02:45:00.350282 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.350204 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:00.352729 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.352695 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq\" (UID: \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:00.353061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.352916 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9q6\" (UniqueName: \"kubernetes.io/projected/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-kube-api-access-sc9q6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq\" (UID: \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:00.353061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.353032 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqp4j\" (UniqueName: \"kubernetes.io/projected/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-kube-api-access-kqp4j\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf\" (UID: \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:00.353215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.353123 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf\" (UID: \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:00.353424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.353403 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq"] Oct 30 02:45:00.353568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.353522 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:00.372104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.372065 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq"] Oct 30 02:45:00.379828 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.379794 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g"] Oct 30 02:45:00.454713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.454674 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9q6\" (UniqueName: \"kubernetes.io/projected/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-kube-api-access-sc9q6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq\" (UID: \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:00.454897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.454724 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c42c5-d6c3-4f7c-afc2-14f672550351-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g\" (UID: \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:00.454897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.454782 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqp4j\" (UniqueName: \"kubernetes.io/projected/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-kube-api-access-kqp4j\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf\" (UID: \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:00.454897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.454856 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4kw\" (UniqueName: \"kubernetes.io/projected/dc8c42c5-d6c3-4f7c-afc2-14f672550351-kube-api-access-wc4kw\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g\" (UID: \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:00.455020 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.454976 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf\" (UID: \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:00.455074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.455062 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jpn5\" (UniqueName: \"kubernetes.io/projected/952801bc-170d-499d-83e1-781126e7cda2-kube-api-access-6jpn5\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq\" (UID: \"952801bc-170d-499d-83e1-781126e7cda2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:00.455113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.455099 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq\" (UID: \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:00.455207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.455188 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952801bc-170d-499d-83e1-781126e7cda2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq\" (UID: \"952801bc-170d-499d-83e1-781126e7cda2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:00.455693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.455670 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf\" (UID: \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:00.455749 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.455731 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq\" (UID: \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:00.477829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.477793 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqp4j\" (UniqueName: \"kubernetes.io/projected/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-kube-api-access-kqp4j\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf\" (UID: \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:00.477996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.477880 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9q6\" (UniqueName: \"kubernetes.io/projected/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-kube-api-access-sc9q6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq\" (UID: \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:00.546090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.546049 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:00.555953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.555908 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4kw\" (UniqueName: \"kubernetes.io/projected/dc8c42c5-d6c3-4f7c-afc2-14f672550351-kube-api-access-wc4kw\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g\" (UID: \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:00.556120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.556036 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jpn5\" (UniqueName: \"kubernetes.io/projected/952801bc-170d-499d-83e1-781126e7cda2-kube-api-access-6jpn5\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq\" (UID: \"952801bc-170d-499d-83e1-781126e7cda2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:00.556169 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.556133 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952801bc-170d-499d-83e1-781126e7cda2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq\" (UID: \"952801bc-170d-499d-83e1-781126e7cda2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:00.556209 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.556198 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c42c5-d6c3-4f7c-afc2-14f672550351-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g\" (UID: \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:00.556806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.556784 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952801bc-170d-499d-83e1-781126e7cda2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq\" (UID: \"952801bc-170d-499d-83e1-781126e7cda2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:00.557022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.556996 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c42c5-d6c3-4f7c-afc2-14f672550351-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g\" (UID: \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:00.579055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.579018 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jpn5\" (UniqueName: \"kubernetes.io/projected/952801bc-170d-499d-83e1-781126e7cda2-kube-api-access-6jpn5\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq\" (UID: \"952801bc-170d-499d-83e1-781126e7cda2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:00.579372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.579344 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4kw\" (UniqueName: \"kubernetes.io/projected/dc8c42c5-d6c3-4f7c-afc2-14f672550351-kube-api-access-wc4kw\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g\" (UID: \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:00.607063 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.607024 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:00.673697 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.673657 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:00.686060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.686015 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:00.820173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.817566 5035 generic.go:356] "Generic (PLEG): container finished" podID="b87fcf2c-3bf3-43be-a5d7-b36ffba88f31" containerID="cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342" exitCode=137 Oct 30 02:45:00.820173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.817730 5035 scope.go:122] "RemoveContainer" containerID="cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342" Oct 30 02:45:00.820173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.817848 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-102e4f" Oct 30 02:45:00.832204 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.832039 5035 scope.go:122] "RemoveContainer" containerID="cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342" Oct 30 02:45:00.832405 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:00.832381 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342\": container with ID starting with cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342 not found: ID does not exist" containerID="cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342" Oct 30 02:45:00.832461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.832415 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342"} err="failed to get container status \"cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342\": rpc error: code = NotFound desc = could not find container \"cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342\": container with ID starting with cd4d8877dff3db639ca66dd44d1ce845cbc6aa9d505b3e9a5f6d354db7a82342 not found: ID does not exist" Oct 30 02:45:00.845966 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:00.845652 5035 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11184-102e4f\" is forbidden: User \"system:node:ip-10-0-142-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-110.ec2.internal' and this object" podUID="b87fcf2c-3bf3-43be-a5d7-b36ffba88f31" pod="opendatahub/test-curl-11184-102e4f" Oct 30 02:45:00.862598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.858549 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf"] Oct 30 02:45:00.923136 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:00.923107 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq"] Oct 30 02:45:00.924333 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:45:00.924304 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bd4e0e_ab0b_4804_966f_a79dbeb3d720.slice/crio-137fee4c4b4303ad633449d78a68454adb5ab9dfb7bcab211dbda1f6f1d6adf0 WatchSource:0}: Error finding container 137fee4c4b4303ad633449d78a68454adb5ab9dfb7bcab211dbda1f6f1d6adf0: Status 404 returned error can't find the container with id 137fee4c4b4303ad633449d78a68454adb5ab9dfb7bcab211dbda1f6f1d6adf0 Oct 30 02:45:01.007534 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.007492 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq"] Oct 30 02:45:01.020326 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.020297 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g"] Oct 30 02:45:01.069772 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:45:01.069738 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952801bc_170d_499d_83e1_781126e7cda2.slice/crio-b62cb9d73a039a84fbf6d693272515515445ec75aa2a1d6354b0ce9165c012dc WatchSource:0}: Error finding container b62cb9d73a039a84fbf6d693272515515445ec75aa2a1d6354b0ce9165c012dc: Status 404 returned error can't find the container with id b62cb9d73a039a84fbf6d693272515515445ec75aa2a1d6354b0ce9165c012dc Oct 30 02:45:01.071387 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:45:01.071357 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8c42c5_d6c3_4f7c_afc2_14f672550351.slice/crio-72fa733e02bdb20cb0eff6d76047fb75b3196f3275057b0d6f18658000581f35 WatchSource:0}: Error finding container 72fa733e02bdb20cb0eff6d76047fb75b3196f3275057b0d6f18658000581f35: Status 404 returned error can't find the container with id 72fa733e02bdb20cb0eff6d76047fb75b3196f3275057b0d6f18658000581f35 Oct 30 02:45:01.645164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.645064 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87fcf2c-3bf3-43be-a5d7-b36ffba88f31" path="/var/lib/kubelet/pods/b87fcf2c-3bf3-43be-a5d7-b36ffba88f31/volumes" Oct 30 02:45:01.652204 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:01.652156 5035 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11184-102e4f\" is forbidden: User \"system:node:ip-10-0-142-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-110.ec2.internal' and this object" podUID="b87fcf2c-3bf3-43be-a5d7-b36ffba88f31" pod="opendatahub/test-curl-11184-102e4f" Oct 30 02:45:01.836562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.836497 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" event={"ID":"952801bc-170d-499d-83e1-781126e7cda2","Type":"ContainerStarted","Data":"2bb78409f8cf809af9b8eac66ddf72bf3a13d42d6373cf293130a1245ece3b86"} Oct 30 02:45:01.836913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.836616 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" event={"ID":"952801bc-170d-499d-83e1-781126e7cda2","Type":"ContainerStarted","Data":"b62cb9d73a039a84fbf6d693272515515445ec75aa2a1d6354b0ce9165c012dc"} Oct 30 02:45:01.838288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.838256 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" event={"ID":"c65263ba-75ac-4cd2-b89e-dbf5bd35988f","Type":"ContainerStarted","Data":"fb92413ec232b6d73a65e00e3cce0ae4185c7487ab92428695632c51ad1f458b"} Oct 30 02:45:01.838420 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.838314 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" event={"ID":"c65263ba-75ac-4cd2-b89e-dbf5bd35988f","Type":"ContainerStarted","Data":"5d385242b7f5fd95cc3cf5fcdcebb889a67bdfa2c164be2f42e7de6d9ed0048c"} Oct 30 02:45:01.839850 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.839823 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" event={"ID":"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720","Type":"ContainerStarted","Data":"904c785afc232c311062410e1ae537669bbc729d8995a6c223ad1d72537d9fc9"} Oct 30 02:45:01.839973 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.839869 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" event={"ID":"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720","Type":"ContainerStarted","Data":"137fee4c4b4303ad633449d78a68454adb5ab9dfb7bcab211dbda1f6f1d6adf0"} Oct 30 02:45:01.842320 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.842292 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" event={"ID":"dc8c42c5-d6c3-4f7c-afc2-14f672550351","Type":"ContainerStarted","Data":"6f13cf2d2a0ee20eaf6438cafa4aed94b45a6b3db75d3cdcdc2396ce9f89b46f"} Oct 30 02:45:01.842437 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.842338 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" event={"ID":"dc8c42c5-d6c3-4f7c-afc2-14f672550351","Type":"ContainerStarted","Data":"72fa733e02bdb20cb0eff6d76047fb75b3196f3275057b0d6f18658000581f35"} Oct 30 02:45:01.877880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.877822 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" podStartSLOduration=1.8778085610000002 podStartE2EDuration="1.877808561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:01.876985152 +0000 UTC m=+4580.379917385" watchObservedRunningTime="2026-10-30 02:45:01.877808561 +0000 UTC m=+4580.380740803" Oct 30 02:45:01.917200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.917083 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" podStartSLOduration=1.917067432 podStartE2EDuration="1.917067432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:01.915988155 +0000 UTC m=+4580.418920376" watchObservedRunningTime="2026-10-30 02:45:01.917067432 +0000 UTC m=+4580.419999703" Oct 30 02:45:01.955136 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.955067 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" podStartSLOduration=1.955055918 podStartE2EDuration="1.955055918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:01.954877739 +0000 UTC m=+4580.457809967" watchObservedRunningTime="2026-10-30 02:45:01.955055918 +0000 UTC m=+4580.457988137" Oct 30 02:45:01.995739 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:01.995675 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" podStartSLOduration=1.995654246 podStartE2EDuration="1.995654246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:01.993847869 +0000 UTC m=+4580.496780110" watchObservedRunningTime="2026-10-30 02:45:01.995654246 +0000 UTC m=+4580.498586462" Oct 30 02:45:02.851793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.851753 5035 generic.go:356] "Generic (PLEG): container finished" podID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerID="904c785afc232c311062410e1ae537669bbc729d8995a6c223ad1d72537d9fc9" exitCode=60 Oct 30 02:45:02.852278 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.851838 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" event={"ID":"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720","Type":"ContainerDied","Data":"904c785afc232c311062410e1ae537669bbc729d8995a6c223ad1d72537d9fc9"} Oct 30 02:45:02.852278 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.852174 5035 scope.go:122] "RemoveContainer" containerID="904c785afc232c311062410e1ae537669bbc729d8995a6c223ad1d72537d9fc9" Oct 30 02:45:02.853632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.853607 5035 generic.go:356] "Generic (PLEG): container finished" podID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerID="6f13cf2d2a0ee20eaf6438cafa4aed94b45a6b3db75d3cdcdc2396ce9f89b46f" exitCode=60 Oct 30 02:45:02.853749 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.853734 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" event={"ID":"dc8c42c5-d6c3-4f7c-afc2-14f672550351","Type":"ContainerDied","Data":"6f13cf2d2a0ee20eaf6438cafa4aed94b45a6b3db75d3cdcdc2396ce9f89b46f"} Oct 30 02:45:02.853956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.853932 5035 scope.go:122] "RemoveContainer" containerID="6f13cf2d2a0ee20eaf6438cafa4aed94b45a6b3db75d3cdcdc2396ce9f89b46f" Oct 30 02:45:02.855753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.855724 5035 generic.go:356] "Generic (PLEG): container finished" podID="952801bc-170d-499d-83e1-781126e7cda2" containerID="2bb78409f8cf809af9b8eac66ddf72bf3a13d42d6373cf293130a1245ece3b86" exitCode=60 Oct 30 02:45:02.855869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.855813 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" event={"ID":"952801bc-170d-499d-83e1-781126e7cda2","Type":"ContainerDied","Data":"2bb78409f8cf809af9b8eac66ddf72bf3a13d42d6373cf293130a1245ece3b86"} Oct 30 02:45:02.856128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.856056 5035 scope.go:122] "RemoveContainer" containerID="2bb78409f8cf809af9b8eac66ddf72bf3a13d42d6373cf293130a1245ece3b86" Oct 30 02:45:02.857779 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.857760 5035 generic.go:356] "Generic (PLEG): container finished" podID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerID="fb92413ec232b6d73a65e00e3cce0ae4185c7487ab92428695632c51ad1f458b" exitCode=60 Oct 30 02:45:02.857848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.857795 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" event={"ID":"c65263ba-75ac-4cd2-b89e-dbf5bd35988f","Type":"ContainerDied","Data":"fb92413ec232b6d73a65e00e3cce0ae4185c7487ab92428695632c51ad1f458b"} Oct 30 02:45:02.858019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:02.857997 5035 scope.go:122] "RemoveContainer" containerID="fb92413ec232b6d73a65e00e3cce0ae4185c7487ab92428695632c51ad1f458b" Oct 30 02:45:03.495451 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.495412 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck"] Oct 30 02:45:03.500162 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.500135 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.523037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.522996 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-584ec5-openshift-default-dockercfg-7kbq5\"" Oct 30 02:45:03.527727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.527690 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck"] Oct 30 02:45:03.601438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.601395 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-token\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.601648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.601449 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln97n\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-kube-api-access-ln97n\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.601648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.601591 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.601729 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.601650 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-credential-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.601787 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.601766 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-certs\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.601827 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.601806 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-data\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.601869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.601841 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-podinfo\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.601957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.601944 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-envoy\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.602017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.602006 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea133939-45a3-47f2-886b-62ebad16f9a3-istiod-ca-cert\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.703719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.703674 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea133939-45a3-47f2-886b-62ebad16f9a3-istiod-ca-cert\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.703719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.703721 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-token\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.703765 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln97n\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-kube-api-access-ln97n\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.703829 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.703882 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-credential-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704015 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.703970 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-certs\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.704101 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-data\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.704184 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-podinfo\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.704280 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-envoy\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.704286 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.704316 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-certs\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.704361 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-credential-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704634 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.704578 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-data\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.704780 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.704755 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea133939-45a3-47f2-886b-62ebad16f9a3-istiod-ca-cert\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.706655 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.706627 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-envoy\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.706782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.706769 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-podinfo\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.727919 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.727879 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-token\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.728119 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.728054 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln97n\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-kube-api-access-ln97n\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.815049 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.815013 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:03.870591 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.870551 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" event={"ID":"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720","Type":"ContainerStarted","Data":"8f3079e90327a1d41d0768137dac507c060c865d5f43a28f4927457d6bf149e3"} Oct 30 02:45:03.887713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.883831 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" event={"ID":"dc8c42c5-d6c3-4f7c-afc2-14f672550351","Type":"ContainerStarted","Data":"4779c1ff874ebc8a7dc48b1cb072568c427ca4ace2832c59b3c38283f6525730"} Oct 30 02:45:03.887713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.886764 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" event={"ID":"952801bc-170d-499d-83e1-781126e7cda2","Type":"ContainerStarted","Data":"3aa614c0ca7dbe9832d867ccd9c9db487720b971c84a302afecdffdb0322635c"} Oct 30 02:45:03.889561 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.889124 5035 generic.go:356] "Generic (PLEG): container finished" podID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerID="b72e7ec4114a4077e2d29560f40bafce8e48631b7c83c234a6ce989a59962e4e" exitCode=60 Oct 30 02:45:03.889561 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.889192 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" event={"ID":"c65263ba-75ac-4cd2-b89e-dbf5bd35988f","Type":"ContainerDied","Data":"b72e7ec4114a4077e2d29560f40bafce8e48631b7c83c234a6ce989a59962e4e"} Oct 30 02:45:03.889561 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.889246 5035 scope.go:122] "RemoveContainer" containerID="fb92413ec232b6d73a65e00e3cce0ae4185c7487ab92428695632c51ad1f458b" Oct 30 02:45:03.889843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:03.889792 5035 scope.go:122] "RemoveContainer" containerID="b72e7ec4114a4077e2d29560f40bafce8e48631b7c83c234a6ce989a59962e4e" Oct 30 02:45:03.890210 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:03.890179 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf_odh-ai-gateway-infra(c65263ba-75ac-4cd2-b89e-dbf5bd35988f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" Oct 30 02:45:04.152629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.152599 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck"] Oct 30 02:45:04.154554 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:45:04.154499 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea133939_45a3_47f2_886b_62ebad16f9a3.slice/crio-0cb914d82615f6e210ded43ecda2a32f457c418be7da3b88697ef8d1a3cfefcd WatchSource:0}: Error finding container 0cb914d82615f6e210ded43ecda2a32f457c418be7da3b88697ef8d1a3cfefcd: Status 404 returned error can't find the container with id 0cb914d82615f6e210ded43ecda2a32f457c418be7da3b88697ef8d1a3cfefcd Oct 30 02:45:04.157446 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.157411 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:45:04.157559 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.157483 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:45:04.157559 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.157517 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:45:04.901674 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.901632 5035 generic.go:356] "Generic (PLEG): container finished" podID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerID="8f3079e90327a1d41d0768137dac507c060c865d5f43a28f4927457d6bf149e3" exitCode=60 Oct 30 02:45:04.902139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.901725 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" event={"ID":"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720","Type":"ContainerDied","Data":"8f3079e90327a1d41d0768137dac507c060c865d5f43a28f4927457d6bf149e3"} Oct 30 02:45:04.902139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.901795 5035 scope.go:122] "RemoveContainer" containerID="904c785afc232c311062410e1ae537669bbc729d8995a6c223ad1d72537d9fc9" Oct 30 02:45:04.902139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.902042 5035 scope.go:122] "RemoveContainer" containerID="8f3079e90327a1d41d0768137dac507c060c865d5f43a28f4927457d6bf149e3" Oct 30 02:45:04.902414 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:04.902391 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq_odh-ai-gateway-infra(c8bd4e0e-ab0b-4804-966f-a79dbeb3d720)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" Oct 30 02:45:04.903980 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.903957 5035 generic.go:356] "Generic (PLEG): container finished" podID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerID="4779c1ff874ebc8a7dc48b1cb072568c427ca4ace2832c59b3c38283f6525730" exitCode=60 Oct 30 02:45:04.904071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.904036 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" event={"ID":"dc8c42c5-d6c3-4f7c-afc2-14f672550351","Type":"ContainerDied","Data":"4779c1ff874ebc8a7dc48b1cb072568c427ca4ace2832c59b3c38283f6525730"} Oct 30 02:45:04.904313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.904293 5035 scope.go:122] "RemoveContainer" containerID="4779c1ff874ebc8a7dc48b1cb072568c427ca4ace2832c59b3c38283f6525730" Oct 30 02:45:04.904684 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:04.904665 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g_odh-ai-gateway-infra(dc8c42c5-d6c3-4f7c-afc2-14f672550351)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" Oct 30 02:45:04.905977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.905903 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" event={"ID":"ea133939-45a3-47f2-886b-62ebad16f9a3","Type":"ContainerStarted","Data":"49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894"} Oct 30 02:45:04.905977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.905952 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" event={"ID":"ea133939-45a3-47f2-886b-62ebad16f9a3","Type":"ContainerStarted","Data":"0cb914d82615f6e210ded43ecda2a32f457c418be7da3b88697ef8d1a3cfefcd"} Oct 30 02:45:04.908143 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.908111 5035 generic.go:356] "Generic (PLEG): container finished" podID="952801bc-170d-499d-83e1-781126e7cda2" containerID="3aa614c0ca7dbe9832d867ccd9c9db487720b971c84a302afecdffdb0322635c" exitCode=60 Oct 30 02:45:04.908233 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.908171 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" event={"ID":"952801bc-170d-499d-83e1-781126e7cda2","Type":"ContainerDied","Data":"3aa614c0ca7dbe9832d867ccd9c9db487720b971c84a302afecdffdb0322635c"} Oct 30 02:45:04.908501 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.908480 5035 scope.go:122] "RemoveContainer" containerID="3aa614c0ca7dbe9832d867ccd9c9db487720b971c84a302afecdffdb0322635c" Oct 30 02:45:04.908870 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:04.908847 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq_odh-ai-gateway-infra(952801bc-170d-499d-83e1-781126e7cda2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" podUID="952801bc-170d-499d-83e1-781126e7cda2" Oct 30 02:45:04.910168 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.910151 5035 scope.go:122] "RemoveContainer" containerID="b72e7ec4114a4077e2d29560f40bafce8e48631b7c83c234a6ce989a59962e4e" Oct 30 02:45:04.910410 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:04.910393 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf_odh-ai-gateway-infra(c65263ba-75ac-4cd2-b89e-dbf5bd35988f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" Oct 30 02:45:04.915908 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.915888 5035 scope.go:122] "RemoveContainer" containerID="6f13cf2d2a0ee20eaf6438cafa4aed94b45a6b3db75d3cdcdc2396ce9f89b46f" Oct 30 02:45:04.929412 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:04.929383 5035 scope.go:122] "RemoveContainer" containerID="2bb78409f8cf809af9b8eac66ddf72bf3a13d42d6373cf293130a1245ece3b86" Oct 30 02:45:05.101455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.101381 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" podStartSLOduration=2.101356974 podStartE2EDuration="2.101356974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:05.100159087 +0000 UTC m=+4583.603091309" watchObservedRunningTime="2026-10-30 02:45:05.101356974 +0000 UTC m=+4583.604289199" Oct 30 02:45:05.554384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.554343 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk"] Oct 30 02:45:05.559769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.559746 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.584559 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.584314 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:45:05.601502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.600953 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk"] Oct 30 02:45:05.735447 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.735404 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-913e8435-sgpkk\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.735684 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.735655 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-857vp\" (UniqueName: \"kubernetes.io/projected/ea090139-df87-41c5-a7ca-6880c9ef8d8d-kube-api-access-857vp\") pod \"maas-api-revoke-sub-913e8435-sgpkk\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.735736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.735710 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-913e8435-sgpkk\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.816154 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.816052 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:05.817762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.817733 5035 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 02:45:05.817896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.817780 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 02:45:05.836764 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.836726 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-857vp\" (UniqueName: \"kubernetes.io/projected/ea090139-df87-41c5-a7ca-6880c9ef8d8d-kube-api-access-857vp\") pod \"maas-api-revoke-sub-913e8435-sgpkk\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.836951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.836776 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-913e8435-sgpkk\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.836951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.836876 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-913e8435-sgpkk\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.837597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.837577 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-913e8435-sgpkk\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.839439 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.839408 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-913e8435-sgpkk\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.887555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.881013 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-857vp\" (UniqueName: \"kubernetes.io/projected/ea090139-df87-41c5-a7ca-6880c9ef8d8d-kube-api-access-857vp\") pod \"maas-api-revoke-sub-913e8435-sgpkk\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.906222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.906189 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:05.919214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.919181 5035 scope.go:122] "RemoveContainer" containerID="3aa614c0ca7dbe9832d867ccd9c9db487720b971c84a302afecdffdb0322635c" Oct 30 02:45:05.919617 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:05.919585 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq_odh-ai-gateway-infra(952801bc-170d-499d-83e1-781126e7cda2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" podUID="952801bc-170d-499d-83e1-781126e7cda2" Oct 30 02:45:05.924395 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.924220 5035 scope.go:122] "RemoveContainer" containerID="8f3079e90327a1d41d0768137dac507c060c865d5f43a28f4927457d6bf149e3" Oct 30 02:45:05.924640 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:05.924605 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq_odh-ai-gateway-infra(c8bd4e0e-ab0b-4804-966f-a79dbeb3d720)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" Oct 30 02:45:05.926598 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:05.926577 5035 scope.go:122] "RemoveContainer" containerID="4779c1ff874ebc8a7dc48b1cb072568c427ca4ace2832c59b3c38283f6525730" Oct 30 02:45:05.926883 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:05.926864 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g_odh-ai-gateway-infra(dc8c42c5-d6c3-4f7c-afc2-14f672550351)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" Oct 30 02:45:06.297877 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:06.297837 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk"] Oct 30 02:45:06.815462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:06.815426 5035 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 02:45:06.815673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:06.815482 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 02:45:06.937520 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:06.937465 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" event={"ID":"ea090139-df87-41c5-a7ca-6880c9ef8d8d","Type":"ContainerStarted","Data":"bf326ac648e7b3ddd1b7a9e885b10a4f0cb2dd3d297c0bcc7983319bd25c5dc2"} Oct 30 02:45:06.938004 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:06.937594 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" event={"ID":"ea090139-df87-41c5-a7ca-6880c9ef8d8d","Type":"ContainerStarted","Data":"654e40c88744b038a2a14d4317e1db2c54f1d7d98e4fa1a3da68f1c749f49065"} Oct 30 02:45:06.996557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:06.992960 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" podStartSLOduration=1.992938626 podStartE2EDuration="1.992938626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:06.989213846 +0000 UTC m=+4585.492146092" watchObservedRunningTime="2026-10-30 02:45:06.992938626 +0000 UTC m=+4585.495870847" Oct 30 02:45:07.816311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:07.816212 5035 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 02:45:07.816311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:07.816279 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 02:45:07.946503 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:07.946464 5035 generic.go:356] "Generic (PLEG): container finished" podID="ea090139-df87-41c5-a7ca-6880c9ef8d8d" containerID="bf326ac648e7b3ddd1b7a9e885b10a4f0cb2dd3d297c0bcc7983319bd25c5dc2" exitCode=0 Oct 30 02:45:07.946967 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:07.946559 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" event={"ID":"ea090139-df87-41c5-a7ca-6880c9ef8d8d","Type":"ContainerDied","Data":"bf326ac648e7b3ddd1b7a9e885b10a4f0cb2dd3d297c0bcc7983319bd25c5dc2"} Oct 30 02:45:08.815664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:08.815622 5035 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 02:45:08.815849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:08.815683 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 02:45:09.200469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.200433 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:09.308073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.307956 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea090139-df87-41c5-a7ca-6880c9ef8d8d-kube-api-access-857vp\" (UniqueName: \"kubernetes.io/projected/ea090139-df87-41c5-a7ca-6880c9ef8d8d-kube-api-access-857vp\") pod \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " Oct 30 02:45:09.308643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.308207 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-xks-ca\") pod \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " Oct 30 02:45:09.308643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.308555 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-openshift-ca\") pod \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\" (UID: \"ea090139-df87-41c5-a7ca-6880c9ef8d8d\") " Oct 30 02:45:09.309490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.309426 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-openshift-ca" pod "ea090139-df87-41c5-a7ca-6880c9ef8d8d" (UID: "ea090139-df87-41c5-a7ca-6880c9ef8d8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:09.309914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.309875 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:09.312310 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.312232 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-xks-ca" pod "ea090139-df87-41c5-a7ca-6880c9ef8d8d" (UID: "ea090139-df87-41c5-a7ca-6880c9ef8d8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:09.312459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.312396 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea090139-df87-41c5-a7ca-6880c9ef8d8d-kube-api-access-857vp" pod "ea090139-df87-41c5-a7ca-6880c9ef8d8d" (UID: "ea090139-df87-41c5-a7ca-6880c9ef8d8d"). InnerVolumeSpecName "kube-api-access-857vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:09.411854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.411758 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea090139-df87-41c5-a7ca-6880c9ef8d8d-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:09.411854 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.411797 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-857vp\" (UniqueName: \"kubernetes.io/projected/ea090139-df87-41c5-a7ca-6880c9ef8d8d-kube-api-access-857vp\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:09.816269 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.816222 5035 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 30 02:45:09.816497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.816280 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 30 02:45:09.998485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.998409 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" Oct 30 02:45:09.998885 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.998407 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk" event={"ID":"ea090139-df87-41c5-a7ca-6880c9ef8d8d","Type":"ContainerDied","Data":"654e40c88744b038a2a14d4317e1db2c54f1d7d98e4fa1a3da68f1c749f49065"} Oct 30 02:45:09.998885 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:09.998617 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="654e40c88744b038a2a14d4317e1db2c54f1d7d98e4fa1a3da68f1c749f49065" Oct 30 02:45:10.826006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:10.825969 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:10.826523 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:10.826442 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:10.827341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:10.827315 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:45:14.635180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:14.635139 5035 scope.go:122] "RemoveContainer" containerID="8f3079e90327a1d41d0768137dac507c060c865d5f43a28f4927457d6bf149e3" Oct 30 02:45:14.635692 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:14.635232 5035 scope.go:122] "RemoveContainer" containerID="3aa614c0ca7dbe9832d867ccd9c9db487720b971c84a302afecdffdb0322635c" Oct 30 02:45:16.046354 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:16.046313 5035 generic.go:356] "Generic (PLEG): container finished" podID="952801bc-170d-499d-83e1-781126e7cda2" containerID="01a3480b626987352712f832faecc728f0b4005336b335379df24cd1b641ec74" exitCode=60 Oct 30 02:45:16.046836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:16.046382 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" event={"ID":"952801bc-170d-499d-83e1-781126e7cda2","Type":"ContainerDied","Data":"01a3480b626987352712f832faecc728f0b4005336b335379df24cd1b641ec74"} Oct 30 02:45:16.046836 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:16.046447 5035 scope.go:122] "RemoveContainer" containerID="3aa614c0ca7dbe9832d867ccd9c9db487720b971c84a302afecdffdb0322635c" Oct 30 02:45:16.046944 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:16.046913 5035 scope.go:122] "RemoveContainer" containerID="01a3480b626987352712f832faecc728f0b4005336b335379df24cd1b641ec74" Oct 30 02:45:16.047316 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:16.047288 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq_odh-ai-gateway-infra(952801bc-170d-499d-83e1-781126e7cda2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" podUID="952801bc-170d-499d-83e1-781126e7cda2" Oct 30 02:45:16.048736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:16.048712 5035 generic.go:356] "Generic (PLEG): container finished" podID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerID="90b2a0e17fb22fd9fe701677120ee41490b2a2bebd67cfe9c8abd05c713ffd1b" exitCode=60 Oct 30 02:45:16.048828 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:16.048781 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" event={"ID":"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720","Type":"ContainerDied","Data":"90b2a0e17fb22fd9fe701677120ee41490b2a2bebd67cfe9c8abd05c713ffd1b"} Oct 30 02:45:16.049168 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:16.049154 5035 scope.go:122] "RemoveContainer" containerID="90b2a0e17fb22fd9fe701677120ee41490b2a2bebd67cfe9c8abd05c713ffd1b" Oct 30 02:45:16.049467 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:16.049451 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq_odh-ai-gateway-infra(c8bd4e0e-ab0b-4804-966f-a79dbeb3d720)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" Oct 30 02:45:16.060201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:16.060181 5035 scope.go:122] "RemoveContainer" containerID="8f3079e90327a1d41d0768137dac507c060c865d5f43a28f4927457d6bf149e3" Oct 30 02:45:17.100689 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.100646 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq"] Oct 30 02:45:17.144907 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.144866 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq"] Oct 30 02:45:17.296954 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.296857 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:17.355462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.355390 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:17.419497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.419447 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-maas-api-service-ca\") pod \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\" (UID: \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\") " Oct 30 02:45:17.419717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.419595 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-kube-api-access-sc9q6\" (UniqueName: \"kubernetes.io/projected/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-kube-api-access-sc9q6\") pod \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\" (UID: \"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720\") " Oct 30 02:45:17.419717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.419628 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952801bc-170d-499d-83e1-781126e7cda2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952801bc-170d-499d-83e1-781126e7cda2-maas-api-service-ca\") pod \"952801bc-170d-499d-83e1-781126e7cda2\" (UID: \"952801bc-170d-499d-83e1-781126e7cda2\") " Oct 30 02:45:17.419823 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.419764 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952801bc-170d-499d-83e1-781126e7cda2-kube-api-access-6jpn5\" (UniqueName: \"kubernetes.io/projected/952801bc-170d-499d-83e1-781126e7cda2-kube-api-access-6jpn5\") pod \"952801bc-170d-499d-83e1-781126e7cda2\" (UID: \"952801bc-170d-499d-83e1-781126e7cda2\") " Oct 30 02:45:17.419987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.419955 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-maas-api-service-ca" pod "c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" (UID: "c8bd4e0e-ab0b-4804-966f-a79dbeb3d720"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:17.420261 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.420235 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952801bc-170d-499d-83e1-781126e7cda2-maas-api-service-ca" pod "952801bc-170d-499d-83e1-781126e7cda2" (UID: "952801bc-170d-499d-83e1-781126e7cda2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:17.420471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.420454 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/952801bc-170d-499d-83e1-781126e7cda2-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:17.420586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.420478 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:17.422038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.422017 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-kube-api-access-sc9q6" pod "c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" (UID: "c8bd4e0e-ab0b-4804-966f-a79dbeb3d720"). InnerVolumeSpecName "kube-api-access-sc9q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:17.422196 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.422181 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952801bc-170d-499d-83e1-781126e7cda2-kube-api-access-6jpn5" pod "952801bc-170d-499d-83e1-781126e7cda2" (UID: "952801bc-170d-499d-83e1-781126e7cda2"). InnerVolumeSpecName "kube-api-access-6jpn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:17.522029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.521989 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc9q6\" (UniqueName: \"kubernetes.io/projected/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720-kube-api-access-sc9q6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:17.522029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.522025 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jpn5\" (UniqueName: \"kubernetes.io/projected/952801bc-170d-499d-83e1-781126e7cda2-kube-api-access-6jpn5\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:17.629076 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:17.628985 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:45:17.636608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.636545 5035 scope.go:122] "RemoveContainer" containerID="b72e7ec4114a4077e2d29560f40bafce8e48631b7c83c234a6ce989a59962e4e" Oct 30 02:45:17.636897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:17.636714 5035 scope.go:122] "RemoveContainer" containerID="4779c1ff874ebc8a7dc48b1cb072568c427ca4ace2832c59b3c38283f6525730" Oct 30 02:45:18.068728 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.068693 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" Oct 30 02:45:18.068937 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.068702 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq" event={"ID":"952801bc-170d-499d-83e1-781126e7cda2","Type":"ContainerDied","Data":"b62cb9d73a039a84fbf6d693272515515445ec75aa2a1d6354b0ce9165c012dc"} Oct 30 02:45:18.068937 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.068856 5035 scope.go:122] "RemoveContainer" containerID="01a3480b626987352712f832faecc728f0b4005336b335379df24cd1b641ec74" Oct 30 02:45:18.070774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.070745 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" Oct 30 02:45:18.070901 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.070776 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq" event={"ID":"c8bd4e0e-ab0b-4804-966f-a79dbeb3d720","Type":"ContainerDied","Data":"137fee4c4b4303ad633449d78a68454adb5ab9dfb7bcab211dbda1f6f1d6adf0"} Oct 30 02:45:18.081912 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.081858 5035 scope.go:122] "RemoveContainer" containerID="90b2a0e17fb22fd9fe701677120ee41490b2a2bebd67cfe9c8abd05c713ffd1b" Oct 30 02:45:18.123970 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.123922 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq"] Oct 30 02:45:18.133668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.133622 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888805-6jxpq"] Oct 30 02:45:18.170847 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.170751 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq"] Oct 30 02:45:18.183077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:18.183040 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888805-jljhq"] Oct 30 02:45:19.080072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.080035 5035 generic.go:356] "Generic (PLEG): container finished" podID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerID="f5595d9a273a0196289e446fc3c41ba5fb594b3c34940362a303d40d25270bab" exitCode=60 Oct 30 02:45:19.080264 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.080109 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" event={"ID":"dc8c42c5-d6c3-4f7c-afc2-14f672550351","Type":"ContainerDied","Data":"f5595d9a273a0196289e446fc3c41ba5fb594b3c34940362a303d40d25270bab"} Oct 30 02:45:19.080264 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.080179 5035 scope.go:122] "RemoveContainer" containerID="4779c1ff874ebc8a7dc48b1cb072568c427ca4ace2832c59b3c38283f6525730" Oct 30 02:45:19.080681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.080652 5035 scope.go:122] "RemoveContainer" containerID="f5595d9a273a0196289e446fc3c41ba5fb594b3c34940362a303d40d25270bab" Oct 30 02:45:19.081162 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:19.081107 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g_odh-ai-gateway-infra(dc8c42c5-d6c3-4f7c-afc2-14f672550351)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" Oct 30 02:45:19.084499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.084463 5035 generic.go:356] "Generic (PLEG): container finished" podID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerID="39cf182d80f7cff1c40182779c547ef643fbaf785eb0bb2518f967e2f0cb1325" exitCode=60 Oct 30 02:45:19.084651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.084566 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" event={"ID":"c65263ba-75ac-4cd2-b89e-dbf5bd35988f","Type":"ContainerDied","Data":"39cf182d80f7cff1c40182779c547ef643fbaf785eb0bb2518f967e2f0cb1325"} Oct 30 02:45:19.085033 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.085017 5035 scope.go:122] "RemoveContainer" containerID="39cf182d80f7cff1c40182779c547ef643fbaf785eb0bb2518f967e2f0cb1325" Oct 30 02:45:19.085393 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:19.085374 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf_odh-ai-gateway-infra(c65263ba-75ac-4cd2-b89e-dbf5bd35988f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" Oct 30 02:45:19.094819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.094792 5035 scope.go:122] "RemoveContainer" containerID="b72e7ec4114a4077e2d29560f40bafce8e48631b7c83c234a6ce989a59962e4e" Oct 30 02:45:19.644409 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.644368 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952801bc-170d-499d-83e1-781126e7cda2" path="/var/lib/kubelet/pods/952801bc-170d-499d-83e1-781126e7cda2/volumes" Oct 30 02:45:19.644870 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:19.644855 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" path="/var/lib/kubelet/pods/c8bd4e0e-ab0b-4804-966f-a79dbeb3d720/volumes" Oct 30 02:45:20.131879 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.131839 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g"] Oct 30 02:45:20.173950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.173914 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf"] Oct 30 02:45:20.320690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.320659 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:20.358811 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.358782 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8c42c5-d6c3-4f7c-afc2-14f672550351-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c42c5-d6c3-4f7c-afc2-14f672550351-maas-api-service-ca\") pod \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\" (UID: \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\") " Oct 30 02:45:20.358979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.358851 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8c42c5-d6c3-4f7c-afc2-14f672550351-kube-api-access-wc4kw\" (UniqueName: \"kubernetes.io/projected/dc8c42c5-d6c3-4f7c-afc2-14f672550351-kube-api-access-wc4kw\") pod \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\" (UID: \"dc8c42c5-d6c3-4f7c-afc2-14f672550351\") " Oct 30 02:45:20.359199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.359176 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8c42c5-d6c3-4f7c-afc2-14f672550351-maas-api-service-ca" pod "dc8c42c5-d6c3-4f7c-afc2-14f672550351" (UID: "dc8c42c5-d6c3-4f7c-afc2-14f672550351"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:20.359471 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.359454 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8c42c5-d6c3-4f7c-afc2-14f672550351-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:20.359843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.359820 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:20.361393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.361365 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8c42c5-d6c3-4f7c-afc2-14f672550351-kube-api-access-wc4kw" pod "dc8c42c5-d6c3-4f7c-afc2-14f672550351" (UID: "dc8c42c5-d6c3-4f7c-afc2-14f672550351"). InnerVolumeSpecName "kube-api-access-wc4kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:20.460288 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.460253 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-maas-api-service-ca\") pod \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\" (UID: \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\") " Oct 30 02:45:20.460469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.460325 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-kube-api-access-kqp4j\" (UniqueName: \"kubernetes.io/projected/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-kube-api-access-kqp4j\") pod \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\" (UID: \"c65263ba-75ac-4cd2-b89e-dbf5bd35988f\") " Oct 30 02:45:20.460661 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.460626 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-maas-api-service-ca" pod "c65263ba-75ac-4cd2-b89e-dbf5bd35988f" (UID: "c65263ba-75ac-4cd2-b89e-dbf5bd35988f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:20.460856 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.460830 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4kw\" (UniqueName: \"kubernetes.io/projected/dc8c42c5-d6c3-4f7c-afc2-14f672550351-kube-api-access-wc4kw\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:20.460856 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.460856 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:20.462701 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.462671 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-kube-api-access-kqp4j" pod "c65263ba-75ac-4cd2-b89e-dbf5bd35988f" (UID: "c65263ba-75ac-4cd2-b89e-dbf5bd35988f"). InnerVolumeSpecName "kube-api-access-kqp4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:20.562243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:20.562198 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqp4j\" (UniqueName: \"kubernetes.io/projected/c65263ba-75ac-4cd2-b89e-dbf5bd35988f-kube-api-access-kqp4j\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.107740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.106772 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" event={"ID":"c65263ba-75ac-4cd2-b89e-dbf5bd35988f","Type":"ContainerDied","Data":"5d385242b7f5fd95cc3cf5fcdcebb889a67bdfa2c164be2f42e7de6d9ed0048c"} Oct 30 02:45:21.107740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.106848 5035 scope.go:122] "RemoveContainer" containerID="39cf182d80f7cff1c40182779c547ef643fbaf785eb0bb2518f967e2f0cb1325" Oct 30 02:45:21.107740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.106957 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf" Oct 30 02:45:21.111120 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.111076 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" Oct 30 02:45:21.111355 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.111083 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g" event={"ID":"dc8c42c5-d6c3-4f7c-afc2-14f672550351","Type":"ContainerDied","Data":"72fa733e02bdb20cb0eff6d76047fb75b3196f3275057b0d6f18658000581f35"} Oct 30 02:45:21.132024 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.131990 5035 scope.go:122] "RemoveContainer" containerID="f5595d9a273a0196289e446fc3c41ba5fb594b3c34940362a303d40d25270bab" Oct 30 02:45:21.183977 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.183916 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g"] Oct 30 02:45:21.191692 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.191647 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888805-tnj2g"] Oct 30 02:45:21.238588 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.234305 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf"] Oct 30 02:45:21.238588 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.237731 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888805-b5nwf"] Oct 30 02:45:21.648028 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.647989 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" path="/var/lib/kubelet/pods/c65263ba-75ac-4cd2-b89e-dbf5bd35988f/volumes" Oct 30 02:45:21.648722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:21.648698 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" path="/var/lib/kubelet/pods/dc8c42c5-d6c3-4f7c-afc2-14f672550351/volumes" Oct 30 02:45:29.505806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:29.505765 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm"] Oct 30 02:45:40.014523 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.014481 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2"] Oct 30 02:45:40.015449 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.015409 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" podUID="e7417f30-4ad3-47c8-835d-b0511309cac5" containerName="maas-api" containerID="cri-o://15821988c7a655e7b2a971645068fe0a3e47e9dee2dc2af02bd485ff0531f754" gracePeriod=30 Oct 30 02:45:40.062453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.062413 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" podUID="e7417f30-4ad3-47c8-835d-b0511309cac5" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.208:8443/health\": dial tcp 10.132.0.208:8443: connect: connection refused" Oct 30 02:45:40.271957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.271837 5035 generic.go:356] "Generic (PLEG): container finished" podID="e7417f30-4ad3-47c8-835d-b0511309cac5" containerID="15821988c7a655e7b2a971645068fe0a3e47e9dee2dc2af02bd485ff0531f754" exitCode=0 Oct 30 02:45:40.271957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.271895 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" event={"ID":"e7417f30-4ad3-47c8-835d-b0511309cac5","Type":"ContainerDied","Data":"15821988c7a655e7b2a971645068fe0a3e47e9dee2dc2af02bd485ff0531f754"} Oct 30 02:45:40.315579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.315493 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:45:40.363957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.363104 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7417f30-4ad3-47c8-835d-b0511309cac5-kube-api-access-r7fq7\" (UniqueName: \"kubernetes.io/projected/e7417f30-4ad3-47c8-835d-b0511309cac5-kube-api-access-r7fq7\") pod \"e7417f30-4ad3-47c8-835d-b0511309cac5\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " Oct 30 02:45:40.378579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.364939 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7417f30-4ad3-47c8-835d-b0511309cac5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7417f30-4ad3-47c8-835d-b0511309cac5-openshift-service-ca\") pod \"e7417f30-4ad3-47c8-835d-b0511309cac5\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " Oct 30 02:45:40.378579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.365068 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7417f30-4ad3-47c8-835d-b0511309cac5-openshift-service-ca" pod "e7417f30-4ad3-47c8-835d-b0511309cac5" (UID: "e7417f30-4ad3-47c8-835d-b0511309cac5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:40.378579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.365222 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls\") pod \"e7417f30-4ad3-47c8-835d-b0511309cac5\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " Oct 30 02:45:40.378579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.365370 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-metrics-certs\") pod \"e7417f30-4ad3-47c8-835d-b0511309cac5\" (UID: \"e7417f30-4ad3-47c8-835d-b0511309cac5\") " Oct 30 02:45:40.378579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.366005 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7417f30-4ad3-47c8-835d-b0511309cac5-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:40.378579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.369167 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7417f30-4ad3-47c8-835d-b0511309cac5-kube-api-access-r7fq7" pod "e7417f30-4ad3-47c8-835d-b0511309cac5" (UID: "e7417f30-4ad3-47c8-835d-b0511309cac5"). InnerVolumeSpecName "kube-api-access-r7fq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:40.378579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.375046 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-metrics-certs" pod "e7417f30-4ad3-47c8-835d-b0511309cac5" (UID: "e7417f30-4ad3-47c8-835d-b0511309cac5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:40.378579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.375817 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls" pod "e7417f30-4ad3-47c8-835d-b0511309cac5" (UID: "e7417f30-4ad3-47c8-835d-b0511309cac5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:40.467478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.467438 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:40.467478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.467473 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7417f30-4ad3-47c8-835d-b0511309cac5-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:40.467478 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:40.467482 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7fq7\" (UniqueName: \"kubernetes.io/projected/e7417f30-4ad3-47c8-835d-b0511309cac5-kube-api-access-r7fq7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:41.279853 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:41.279819 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" Oct 30 02:45:41.280346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:41.279854 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2" event={"ID":"e7417f30-4ad3-47c8-835d-b0511309cac5","Type":"ContainerDied","Data":"589c2f1209e91a11e5988cd1182349e7a679572325d50836927e741108ed17a3"} Oct 30 02:45:41.280346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:41.279922 5035 scope.go:122] "RemoveContainer" containerID="15821988c7a655e7b2a971645068fe0a3e47e9dee2dc2af02bd485ff0531f754" Oct 30 02:45:41.353776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:41.353734 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2"] Oct 30 02:45:41.365093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:41.365053 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-95f40f-b5b945c79-sf5p2"] Oct 30 02:45:41.644305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:41.644223 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7417f30-4ad3-47c8-835d-b0511309cac5" path="/var/lib/kubelet/pods/e7417f30-4ad3-47c8-835d-b0511309cac5/volumes" Oct 30 02:45:44.543376 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.543197 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" podUID="2f369a52-a39c-490e-bf37-22bb44483960" containerName="main" containerID="cri-o://cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174" gracePeriod=45 Oct 30 02:45:44.845262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.845202 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:45:44.926227 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.926178 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-tmp-dir\") pod \"2f369a52-a39c-490e-bf37-22bb44483960\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " Oct 30 02:45:44.926447 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.926298 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-home\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-home\") pod \"2f369a52-a39c-490e-bf37-22bb44483960\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " Oct 30 02:45:44.926546 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.926462 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f369a52-a39c-490e-bf37-22bb44483960-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f369a52-a39c-490e-bf37-22bb44483960-tls-certs\") pod \"2f369a52-a39c-490e-bf37-22bb44483960\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " Oct 30 02:45:44.926546 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.926505 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-dshm\") pod \"2f369a52-a39c-490e-bf37-22bb44483960\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " Oct 30 02:45:44.926684 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.926556 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-tmp-dir" pod "2f369a52-a39c-490e-bf37-22bb44483960" (UID: "2f369a52-a39c-490e-bf37-22bb44483960"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:44.926684 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.926631 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-home" pod "2f369a52-a39c-490e-bf37-22bb44483960" (UID: "2f369a52-a39c-490e-bf37-22bb44483960"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:44.926684 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.926664 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f369a52-a39c-490e-bf37-22bb44483960-kube-api-access-568lp\" (UniqueName: \"kubernetes.io/projected/2f369a52-a39c-490e-bf37-22bb44483960-kube-api-access-568lp\") pod \"2f369a52-a39c-490e-bf37-22bb44483960\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " Oct 30 02:45:44.926861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.926750 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-model-cache\") pod \"2f369a52-a39c-490e-bf37-22bb44483960\" (UID: \"2f369a52-a39c-490e-bf37-22bb44483960\") " Oct 30 02:45:44.927038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.927017 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-model-cache" pod "2f369a52-a39c-490e-bf37-22bb44483960" (UID: "2f369a52-a39c-490e-bf37-22bb44483960"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:44.927344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.927322 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:44.927448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.927349 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:44.927448 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.927363 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:44.928952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.928920 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-dshm" pod "2f369a52-a39c-490e-bf37-22bb44483960" (UID: "2f369a52-a39c-490e-bf37-22bb44483960"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:44.929069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.928976 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f369a52-a39c-490e-bf37-22bb44483960-tls-certs" pod "2f369a52-a39c-490e-bf37-22bb44483960" (UID: "2f369a52-a39c-490e-bf37-22bb44483960"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:44.929316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:44.929292 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f369a52-a39c-490e-bf37-22bb44483960-kube-api-access-568lp" pod "2f369a52-a39c-490e-bf37-22bb44483960" (UID: "2f369a52-a39c-490e-bf37-22bb44483960"). InnerVolumeSpecName "kube-api-access-568lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:45.029184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.029147 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f369a52-a39c-490e-bf37-22bb44483960-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:45.029184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.029180 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f369a52-a39c-490e-bf37-22bb44483960-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:45.029184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.029190 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-568lp\" (UniqueName: \"kubernetes.io/projected/2f369a52-a39c-490e-bf37-22bb44483960-kube-api-access-568lp\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:45.312341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.312305 5035 generic.go:356] "Generic (PLEG): container finished" podID="2f369a52-a39c-490e-bf37-22bb44483960" containerID="cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174" exitCode=0 Oct 30 02:45:45.312341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.312349 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" Oct 30 02:45:45.312638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.312350 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" event={"ID":"2f369a52-a39c-490e-bf37-22bb44483960","Type":"ContainerDied","Data":"cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174"} Oct 30 02:45:45.312638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.312467 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm" event={"ID":"2f369a52-a39c-490e-bf37-22bb44483960","Type":"ContainerDied","Data":"0d3b31998c2cbd7c05ddf9bda853c4b30e0e06f1d5cd6ece56946bd668a34a38"} Oct 30 02:45:45.312638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.312495 5035 scope.go:122] "RemoveContainer" containerID="cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174" Oct 30 02:45:45.323489 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.323466 5035 scope.go:122] "RemoveContainer" containerID="cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174" Oct 30 02:45:45.323833 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:45.323808 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174\": container with ID starting with cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174 not found: ID does not exist" containerID="cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174" Oct 30 02:45:45.323891 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.323842 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174"} err="failed to get container status \"cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174\": rpc error: code = NotFound desc = could not find container \"cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174\": container with ID starting with cc2d9f1fde4d58872691358ffe5cd701abd80b273c8e65c2de2cddde138e9174 not found: ID does not exist" Oct 30 02:45:45.366909 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.366867 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm"] Oct 30 02:45:45.378978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.378941 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/rate-test-model-594253-kserve-57496d79c-4trkm"] Oct 30 02:45:45.644765 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:45.644675 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f369a52-a39c-490e-bf37-22bb44483960" path="/var/lib/kubelet/pods/2f369a52-a39c-490e-bf37-22bb44483960/volumes" Oct 30 02:45:47.629121 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:45:47.629082 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:45:55.702509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:55.702419 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4"] Oct 30 02:45:55.731334 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:55.731289 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj"] Oct 30 02:45:57.257678 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.257640 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt"] Oct 30 02:45:57.258159 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.257856 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" podUID="2bb4a780-b282-43f7-9173-d6bf26315614" containerName="maas-api" containerID="cri-o://410a259f6f5067e0ad30c7b043e6b4d97b43eef4282067730e45eada8eca133b" gracePeriod=30 Oct 30 02:45:57.416524 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.416486 5035 generic.go:356] "Generic (PLEG): container finished" podID="2bb4a780-b282-43f7-9173-d6bf26315614" containerID="410a259f6f5067e0ad30c7b043e6b4d97b43eef4282067730e45eada8eca133b" exitCode=0 Oct 30 02:45:57.416713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.416578 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" event={"ID":"2bb4a780-b282-43f7-9173-d6bf26315614","Type":"ContainerDied","Data":"410a259f6f5067e0ad30c7b043e6b4d97b43eef4282067730e45eada8eca133b"} Oct 30 02:45:57.570694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.570665 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:45:57.702443 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.702400 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb4a780-b282-43f7-9173-d6bf26315614-kube-api-access-7ctdn\" (UniqueName: \"kubernetes.io/projected/2bb4a780-b282-43f7-9173-d6bf26315614-kube-api-access-7ctdn\") pod \"2bb4a780-b282-43f7-9173-d6bf26315614\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " Oct 30 02:45:57.702721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.702493 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb4a780-b282-43f7-9173-d6bf26315614-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4a780-b282-43f7-9173-d6bf26315614-openshift-service-ca\") pod \"2bb4a780-b282-43f7-9173-d6bf26315614\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " Oct 30 02:45:57.702721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.702630 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-maas-api-tls\") pod \"2bb4a780-b282-43f7-9173-d6bf26315614\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " Oct 30 02:45:57.702721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.702680 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-metrics-certs\") pod \"2bb4a780-b282-43f7-9173-d6bf26315614\" (UID: \"2bb4a780-b282-43f7-9173-d6bf26315614\") " Oct 30 02:45:57.703604 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.703570 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb4a780-b282-43f7-9173-d6bf26315614-openshift-service-ca" pod "2bb4a780-b282-43f7-9173-d6bf26315614" (UID: "2bb4a780-b282-43f7-9173-d6bf26315614"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:57.706137 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.706101 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-maas-api-tls" pod "2bb4a780-b282-43f7-9173-d6bf26315614" (UID: "2bb4a780-b282-43f7-9173-d6bf26315614"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:57.707614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.707583 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb4a780-b282-43f7-9173-d6bf26315614-kube-api-access-7ctdn" pod "2bb4a780-b282-43f7-9173-d6bf26315614" (UID: "2bb4a780-b282-43f7-9173-d6bf26315614"). InnerVolumeSpecName "kube-api-access-7ctdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:57.710020 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.709977 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-metrics-certs" pod "2bb4a780-b282-43f7-9173-d6bf26315614" (UID: "2bb4a780-b282-43f7-9173-d6bf26315614"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:57.803951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.803842 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctdn\" (UniqueName: \"kubernetes.io/projected/2bb4a780-b282-43f7-9173-d6bf26315614-kube-api-access-7ctdn\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:57.803951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.803886 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb4a780-b282-43f7-9173-d6bf26315614-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:57.803951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.803903 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:57.803951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:57.803919 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb4a780-b282-43f7-9173-d6bf26315614-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:45:58.425133 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:58.425098 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" event={"ID":"2bb4a780-b282-43f7-9173-d6bf26315614","Type":"ContainerDied","Data":"9a68abddd615544d6eb0f53a8bb4684d8b23d61063984579d75f6ace1ceab9e9"} Oct 30 02:45:58.425133 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:58.425135 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt" Oct 30 02:45:58.425736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:58.425163 5035 scope.go:122] "RemoveContainer" containerID="410a259f6f5067e0ad30c7b043e6b4d97b43eef4282067730e45eada8eca133b" Oct 30 02:45:58.479164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:58.479122 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt"] Oct 30 02:45:58.491741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:58.491692 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6b8bfa-598fd9679d-4v6wt"] Oct 30 02:45:59.654089 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:45:59.654051 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb4a780-b282-43f7-9173-d6bf26315614" path="/var/lib/kubelet/pods/2bb4a780-b282-43f7-9173-d6bf26315614/volumes" Oct 30 02:46:00.731673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:00.731596 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" podUID="708b87c9-902e-4882-9b48-1ad131b19385" containerName="payload-pre-processing" containerID="cri-o://31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb" gracePeriod=25 Oct 30 02:46:01.008315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.008284 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:46:01.148664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.148510 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708b87c9-902e-4882-9b48-1ad131b19385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708b87c9-902e-4882-9b48-1ad131b19385-plugins-config-volume\") pod \"708b87c9-902e-4882-9b48-1ad131b19385\" (UID: \"708b87c9-902e-4882-9b48-1ad131b19385\") " Oct 30 02:46:01.148899 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.148777 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708b87c9-902e-4882-9b48-1ad131b19385-kube-api-access-rvfz8\" (UniqueName: \"kubernetes.io/projected/708b87c9-902e-4882-9b48-1ad131b19385-kube-api-access-rvfz8\") pod \"708b87c9-902e-4882-9b48-1ad131b19385\" (UID: \"708b87c9-902e-4882-9b48-1ad131b19385\") " Oct 30 02:46:01.149064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.149022 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708b87c9-902e-4882-9b48-1ad131b19385-plugins-config-volume" pod "708b87c9-902e-4882-9b48-1ad131b19385" (UID: "708b87c9-902e-4882-9b48-1ad131b19385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:01.149421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.149398 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/708b87c9-902e-4882-9b48-1ad131b19385-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:01.151237 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.151206 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708b87c9-902e-4882-9b48-1ad131b19385-kube-api-access-rvfz8" pod "708b87c9-902e-4882-9b48-1ad131b19385" (UID: "708b87c9-902e-4882-9b48-1ad131b19385"). InnerVolumeSpecName "kube-api-access-rvfz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:01.250847 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.250749 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfz8\" (UniqueName: \"kubernetes.io/projected/708b87c9-902e-4882-9b48-1ad131b19385-kube-api-access-rvfz8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:01.407663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.407610 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" podUID="5a90a801-f3b4-439b-b4b5-2a5ece592228" containerName="payload-processing" containerID="cri-o://749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24" gracePeriod=25 Oct 30 02:46:01.449668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.449632 5035 generic.go:356] "Generic (PLEG): container finished" podID="708b87c9-902e-4882-9b48-1ad131b19385" containerID="31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb" exitCode=0 Oct 30 02:46:01.449825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.449678 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" Oct 30 02:46:01.449825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.449716 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" event={"ID":"708b87c9-902e-4882-9b48-1ad131b19385","Type":"ContainerDied","Data":"31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb"} Oct 30 02:46:01.449825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.449779 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj" event={"ID":"708b87c9-902e-4882-9b48-1ad131b19385","Type":"ContainerDied","Data":"e2b2e6e9b6580b036d08057f803a1fd2d15288dc5d6026c03f1ca2bf528e17aa"} Oct 30 02:46:01.449825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.449803 5035 scope.go:122] "RemoveContainer" containerID="31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb" Oct 30 02:46:01.520556 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.520519 5035 scope.go:122] "RemoveContainer" containerID="31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb" Oct 30 02:46:01.520955 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:01.520925 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb\": container with ID starting with 31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb not found: ID does not exist" containerID="31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb" Oct 30 02:46:01.521061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.520962 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb"} err="failed to get container status \"31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb\": rpc error: code = NotFound desc = could not find container \"31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb\": container with ID starting with 31362b0c772b441d39fd2284ea934916c9d3719ce5049c5ea9c1f866f178b0bb not found: ID does not exist" Oct 30 02:46:01.527492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.527441 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj"] Oct 30 02:46:01.530741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.530716 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2c4b44-f4ddfc467-4q2rj"] Oct 30 02:46:01.646691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.646651 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708b87c9-902e-4882-9b48-1ad131b19385" path="/var/lib/kubelet/pods/708b87c9-902e-4882-9b48-1ad131b19385/volumes" Oct 30 02:46:01.669676 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.669649 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:46:01.857948 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.857846 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a90a801-f3b4-439b-b4b5-2a5ece592228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a90a801-f3b4-439b-b4b5-2a5ece592228-plugins-config-volume\") pod \"5a90a801-f3b4-439b-b4b5-2a5ece592228\" (UID: \"5a90a801-f3b4-439b-b4b5-2a5ece592228\") " Oct 30 02:46:01.858458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.857953 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a90a801-f3b4-439b-b4b5-2a5ece592228-kube-api-access-q9xkv\" (UniqueName: \"kubernetes.io/projected/5a90a801-f3b4-439b-b4b5-2a5ece592228-kube-api-access-q9xkv\") pod \"5a90a801-f3b4-439b-b4b5-2a5ece592228\" (UID: \"5a90a801-f3b4-439b-b4b5-2a5ece592228\") " Oct 30 02:46:01.858458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.858349 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a90a801-f3b4-439b-b4b5-2a5ece592228-plugins-config-volume" pod "5a90a801-f3b4-439b-b4b5-2a5ece592228" (UID: "5a90a801-f3b4-439b-b4b5-2a5ece592228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:01.860215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.860191 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a90a801-f3b4-439b-b4b5-2a5ece592228-kube-api-access-q9xkv" pod "5a90a801-f3b4-439b-b4b5-2a5ece592228" (UID: "5a90a801-f3b4-439b-b4b5-2a5ece592228"). InnerVolumeSpecName "kube-api-access-q9xkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:01.959881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.959839 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a90a801-f3b4-439b-b4b5-2a5ece592228-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:01.959881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:01.959873 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9xkv\" (UniqueName: \"kubernetes.io/projected/5a90a801-f3b4-439b-b4b5-2a5ece592228-kube-api-access-q9xkv\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:02.458986 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:02.458945 5035 generic.go:356] "Generic (PLEG): container finished" podID="5a90a801-f3b4-439b-b4b5-2a5ece592228" containerID="749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24" exitCode=0 Oct 30 02:46:02.459220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:02.458991 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" Oct 30 02:46:02.459220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:02.459014 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" event={"ID":"5a90a801-f3b4-439b-b4b5-2a5ece592228","Type":"ContainerDied","Data":"749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24"} Oct 30 02:46:02.459220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:02.459063 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4" event={"ID":"5a90a801-f3b4-439b-b4b5-2a5ece592228","Type":"ContainerDied","Data":"348e56a1ab206e30226262bf14315fefd3ced32eb9ffeef0ea7b6f88dae75a43"} Oct 30 02:46:02.459220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:02.459088 5035 scope.go:122] "RemoveContainer" containerID="749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24" Oct 30 02:46:02.470305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:02.470278 5035 scope.go:122] "RemoveContainer" containerID="749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24" Oct 30 02:46:02.470735 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:02.470686 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24\": container with ID starting with 749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24 not found: ID does not exist" containerID="749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24" Oct 30 02:46:02.470914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:02.470748 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24"} err="failed to get container status \"749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24\": rpc error: code = NotFound desc = could not find container \"749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24\": container with ID starting with 749037c4ee0aea85edba5ec876d30ae0fc4e3fc11a6aebb7db8bcb789cc1fa24 not found: ID does not exist" Oct 30 02:46:02.513440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:02.513401 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4"] Oct 30 02:46:02.525400 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:02.525358 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2c4b44-76d85598d-jznf4"] Oct 30 02:46:03.644740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:03.644702 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a90a801-f3b4-439b-b4b5-2a5ece592228" path="/var/lib/kubelet/pods/5a90a801-f3b4-439b-b4b5-2a5ece592228/volumes" Oct 30 02:46:07.606568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.606513 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb"] Oct 30 02:46:07.607233 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.606806 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" podUID="d20aac51-0f1a-4f75-81d3-33f010361b2e" containerName="maas-api" containerID="cri-o://996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8" gracePeriod=30 Oct 30 02:46:07.912677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.912629 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:46:07.935573 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.935514 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls\") pod \"d20aac51-0f1a-4f75-81d3-33f010361b2e\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " Oct 30 02:46:07.935849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.935592 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20aac51-0f1a-4f75-81d3-33f010361b2e-kube-api-access-hvf6f\" (UniqueName: \"kubernetes.io/projected/d20aac51-0f1a-4f75-81d3-33f010361b2e-kube-api-access-hvf6f\") pod \"d20aac51-0f1a-4f75-81d3-33f010361b2e\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " Oct 30 02:46:07.935849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.935768 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20aac51-0f1a-4f75-81d3-33f010361b2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20aac51-0f1a-4f75-81d3-33f010361b2e-openshift-service-ca\") pod \"d20aac51-0f1a-4f75-81d3-33f010361b2e\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " Oct 30 02:46:07.935849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.935794 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-metrics-certs\") pod \"d20aac51-0f1a-4f75-81d3-33f010361b2e\" (UID: \"d20aac51-0f1a-4f75-81d3-33f010361b2e\") " Oct 30 02:46:07.936196 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.936165 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20aac51-0f1a-4f75-81d3-33f010361b2e-openshift-service-ca" pod "d20aac51-0f1a-4f75-81d3-33f010361b2e" (UID: "d20aac51-0f1a-4f75-81d3-33f010361b2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:07.936360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.936343 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20aac51-0f1a-4f75-81d3-33f010361b2e-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:07.938100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.938064 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls" pod "d20aac51-0f1a-4f75-81d3-33f010361b2e" (UID: "d20aac51-0f1a-4f75-81d3-33f010361b2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:07.938228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.938074 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20aac51-0f1a-4f75-81d3-33f010361b2e-kube-api-access-hvf6f" pod "d20aac51-0f1a-4f75-81d3-33f010361b2e" (UID: "d20aac51-0f1a-4f75-81d3-33f010361b2e"). InnerVolumeSpecName "kube-api-access-hvf6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:07.938228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:07.938151 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-metrics-certs" pod "d20aac51-0f1a-4f75-81d3-33f010361b2e" (UID: "d20aac51-0f1a-4f75-81d3-33f010361b2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:08.037265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.037226 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:08.037265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.037261 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d20aac51-0f1a-4f75-81d3-33f010361b2e-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:08.037265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.037270 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvf6f\" (UniqueName: \"kubernetes.io/projected/d20aac51-0f1a-4f75-81d3-33f010361b2e-kube-api-access-hvf6f\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:08.502840 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.502798 5035 generic.go:356] "Generic (PLEG): container finished" podID="d20aac51-0f1a-4f75-81d3-33f010361b2e" containerID="996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8" exitCode=0 Oct 30 02:46:08.503069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.502851 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" Oct 30 02:46:08.503069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.502881 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" event={"ID":"d20aac51-0f1a-4f75-81d3-33f010361b2e","Type":"ContainerDied","Data":"996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8"} Oct 30 02:46:08.503069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.502953 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb" event={"ID":"d20aac51-0f1a-4f75-81d3-33f010361b2e","Type":"ContainerDied","Data":"ba8b8b1ab113e7a09a9f10399e6f39b47a2a25e812e35775724903c3cb58cd76"} Oct 30 02:46:08.503069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.502994 5035 scope.go:122] "RemoveContainer" containerID="996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8" Oct 30 02:46:08.515298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.515269 5035 scope.go:122] "RemoveContainer" containerID="996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8" Oct 30 02:46:08.515691 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:08.515666 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8\": container with ID starting with 996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8 not found: ID does not exist" containerID="996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8" Oct 30 02:46:08.515773 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.515701 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8"} err="failed to get container status \"996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8\": rpc error: code = NotFound desc = could not find container \"996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8\": container with ID starting with 996f00c7cf67d069deb3c97ec58543acd972f94f927f0c73d3a02df80d2ebdb8 not found: ID does not exist" Oct 30 02:46:08.554685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.554650 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb"] Oct 30 02:46:08.558162 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:08.558130 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2c4b44-586d58dc47-thckb"] Oct 30 02:46:09.646582 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:09.646520 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20aac51-0f1a-4f75-81d3-33f010361b2e" path="/var/lib/kubelet/pods/d20aac51-0f1a-4f75-81d3-33f010361b2e/volumes" Oct 30 02:46:12.209315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:12.209273 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj"] Oct 30 02:46:17.209859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:17.209815 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" podUID="802465ef-c6e6-40eb-b083-c3fa61aed348" containerName="payload-pre-processing" containerID="cri-o://b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1" gracePeriod=25 Oct 30 02:46:17.629568 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:17.629460 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:46:20.527779 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:20.527739 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5"] Oct 30 02:46:20.546091 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:20.546053 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz"] Oct 30 02:46:22.856212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:22.856167 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" podUID="802465ef-c6e6-40eb-b083-c3fa61aed348" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:25.529030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:25.528967 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" podUID="ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" containerName="payload-processing" containerID="cri-o://bec2352d71cf6d8b40f480c2369fa6cc62342d49053711cf6e4147cf13ed0209" gracePeriod=25 Oct 30 02:46:25.546648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:25.546590 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" podUID="f4b57914-820a-470a-bb14-34c53a03840a" containerName="payload-pre-processing" containerID="cri-o://6f741b6659090c153a51a1480df61c017284033e30497216babb87f6b53e6c43" gracePeriod=25 Oct 30 02:46:30.374182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.374134 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9"] Oct 30 02:46:30.374959 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.374923 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" podUID="053375c1-4f44-4ae1-9272-cbd76b067d5a" containerName="maas-api" containerID="cri-o://5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416" gracePeriod=30 Oct 30 02:46:30.681037 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.681005 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:46:30.698435 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.698394 5035 generic.go:356] "Generic (PLEG): container finished" podID="053375c1-4f44-4ae1-9272-cbd76b067d5a" containerID="5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416" exitCode=0 Oct 30 02:46:30.698673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.698443 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" Oct 30 02:46:30.698673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.698471 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" event={"ID":"053375c1-4f44-4ae1-9272-cbd76b067d5a","Type":"ContainerDied","Data":"5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416"} Oct 30 02:46:30.698673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.698523 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9" event={"ID":"053375c1-4f44-4ae1-9272-cbd76b067d5a","Type":"ContainerDied","Data":"924c64b0b864dbc48c8017e65d0e31e6fcd413e89e715596a6499363d930da75"} Oct 30 02:46:30.698673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.698591 5035 scope.go:122] "RemoveContainer" containerID="5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416" Oct 30 02:46:30.709497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.709469 5035 scope.go:122] "RemoveContainer" containerID="5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416" Oct 30 02:46:30.709975 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:30.709929 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416\": container with ID starting with 5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416 not found: ID does not exist" containerID="5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416" Oct 30 02:46:30.710094 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.709979 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416"} err="failed to get container status \"5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416\": rpc error: code = NotFound desc = could not find container \"5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416\": container with ID starting with 5caa4e2bd4ad2367277f9a286949bea0d631d18f623456c96bfcae150d6ce416 not found: ID does not exist" Oct 30 02:46:30.766432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.766380 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053375c1-4f44-4ae1-9272-cbd76b067d5a-kube-api-access-jvm4p\" (UniqueName: \"kubernetes.io/projected/053375c1-4f44-4ae1-9272-cbd76b067d5a-kube-api-access-jvm4p\") pod \"053375c1-4f44-4ae1-9272-cbd76b067d5a\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " Oct 30 02:46:30.767889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.766807 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-maas-api-tls\") pod \"053375c1-4f44-4ae1-9272-cbd76b067d5a\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " Oct 30 02:46:30.768175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.768138 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/053375c1-4f44-4ae1-9272-cbd76b067d5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053375c1-4f44-4ae1-9272-cbd76b067d5a-openshift-service-ca\") pod \"053375c1-4f44-4ae1-9272-cbd76b067d5a\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " Oct 30 02:46:30.768384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.768361 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-metrics-certs\") pod \"053375c1-4f44-4ae1-9272-cbd76b067d5a\" (UID: \"053375c1-4f44-4ae1-9272-cbd76b067d5a\") " Oct 30 02:46:30.770202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.769950 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053375c1-4f44-4ae1-9272-cbd76b067d5a-openshift-service-ca" pod "053375c1-4f44-4ae1-9272-cbd76b067d5a" (UID: "053375c1-4f44-4ae1-9272-cbd76b067d5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:30.772917 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.772878 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053375c1-4f44-4ae1-9272-cbd76b067d5a-kube-api-access-jvm4p" pod "053375c1-4f44-4ae1-9272-cbd76b067d5a" (UID: "053375c1-4f44-4ae1-9272-cbd76b067d5a"). InnerVolumeSpecName "kube-api-access-jvm4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:30.773055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.773006 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-maas-api-tls" pod "053375c1-4f44-4ae1-9272-cbd76b067d5a" (UID: "053375c1-4f44-4ae1-9272-cbd76b067d5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:30.776030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.775939 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-metrics-certs" pod "053375c1-4f44-4ae1-9272-cbd76b067d5a" (UID: "053375c1-4f44-4ae1-9272-cbd76b067d5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:30.870474 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.870432 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvm4p\" (UniqueName: \"kubernetes.io/projected/053375c1-4f44-4ae1-9272-cbd76b067d5a-kube-api-access-jvm4p\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:30.870474 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.870471 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:30.870474 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.870481 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053375c1-4f44-4ae1-9272-cbd76b067d5a-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:30.870772 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:30.870490 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/053375c1-4f44-4ae1-9272-cbd76b067d5a-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:31.048183 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:31.048144 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9"] Oct 30 02:46:31.062992 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:31.062950 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-584ec5-6dcc9cffd7-qjqp9"] Oct 30 02:46:31.225743 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:31.225698 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" podUID="f4b57914-820a-470a-bb14-34c53a03840a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:31.225929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:31.225705 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" podUID="ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:31.654123 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:31.654077 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="053375c1-4f44-4ae1-9272-cbd76b067d5a" path="/var/lib/kubelet/pods/053375c1-4f44-4ae1-9272-cbd76b067d5a/volumes" Oct 30 02:46:32.856328 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:32.856287 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" podUID="802465ef-c6e6-40eb-b083-c3fa61aed348" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:37.489267 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.489188 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:46:37.561253 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.561216 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802465ef-c6e6-40eb-b083-c3fa61aed348-kube-api-access-bgj6q\" (UniqueName: \"kubernetes.io/projected/802465ef-c6e6-40eb-b083-c3fa61aed348-kube-api-access-bgj6q\") pod \"802465ef-c6e6-40eb-b083-c3fa61aed348\" (UID: \"802465ef-c6e6-40eb-b083-c3fa61aed348\") " Oct 30 02:46:37.561465 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.561279 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802465ef-c6e6-40eb-b083-c3fa61aed348-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802465ef-c6e6-40eb-b083-c3fa61aed348-plugins-config-volume\") pod \"802465ef-c6e6-40eb-b083-c3fa61aed348\" (UID: \"802465ef-c6e6-40eb-b083-c3fa61aed348\") " Oct 30 02:46:37.561821 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.561790 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802465ef-c6e6-40eb-b083-c3fa61aed348-plugins-config-volume" pod "802465ef-c6e6-40eb-b083-c3fa61aed348" (UID: "802465ef-c6e6-40eb-b083-c3fa61aed348"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:37.563730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.563698 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802465ef-c6e6-40eb-b083-c3fa61aed348-kube-api-access-bgj6q" pod "802465ef-c6e6-40eb-b083-c3fa61aed348" (UID: "802465ef-c6e6-40eb-b083-c3fa61aed348"). InnerVolumeSpecName "kube-api-access-bgj6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:37.659173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.659136 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9"] Oct 30 02:46:37.662424 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.662391 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgj6q\" (UniqueName: \"kubernetes.io/projected/802465ef-c6e6-40eb-b083-c3fa61aed348-kube-api-access-bgj6q\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:37.662664 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.662636 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/802465ef-c6e6-40eb-b083-c3fa61aed348-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:37.695541 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.695485 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp"] Oct 30 02:46:37.757594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.757476 5035 generic.go:356] "Generic (PLEG): container finished" podID="802465ef-c6e6-40eb-b083-c3fa61aed348" containerID="b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1" exitCode=0 Oct 30 02:46:37.757594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.757559 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" Oct 30 02:46:37.757829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.757628 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" event={"ID":"802465ef-c6e6-40eb-b083-c3fa61aed348","Type":"ContainerDied","Data":"b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1"} Oct 30 02:46:37.757829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.757696 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj" event={"ID":"802465ef-c6e6-40eb-b083-c3fa61aed348","Type":"ContainerDied","Data":"a636a075ba7e5d7f725452ae533195ab9e8734333c0e7dec93da9c9320beb4c5"} Oct 30 02:46:37.757829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.757739 5035 scope.go:122] "RemoveContainer" containerID="b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1" Oct 30 02:46:37.769076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.769040 5035 scope.go:122] "RemoveContainer" containerID="b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1" Oct 30 02:46:37.769461 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:37.769437 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1\": container with ID starting with b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1 not found: ID does not exist" containerID="b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1" Oct 30 02:46:37.769544 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.769473 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1"} err="failed to get container status \"b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1\": rpc error: code = NotFound desc = could not find container \"b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1\": container with ID starting with b83ed336e27f1bff0d5f00aa627a1a8670216332a7ea8d50036c8a31d8298ea1 not found: ID does not exist" Oct 30 02:46:37.811816 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.811772 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj"] Oct 30 02:46:37.816346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.816310 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9dfb3c-7f56b7789f-sxmvj"] Oct 30 02:46:37.950286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.950250 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck"] Oct 30 02:46:37.950504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:37.950471 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" containerID="cri-o://49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894" gracePeriod=30 Oct 30 02:46:39.643007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:39.642966 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802465ef-c6e6-40eb-b083-c3fa61aed348" path="/var/lib/kubelet/pods/802465ef-c6e6-40eb-b083-c3fa61aed348/volumes" Oct 30 02:46:40.829374 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:40.829325 5035 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:46:40.829867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:40.829404 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:46:41.225567 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:41.225511 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" podUID="f4b57914-820a-470a-bb14-34c53a03840a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:41.225806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:41.225522 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" podUID="ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:42.277657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.277616 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5"] Oct 30 02:46:42.278217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.277893 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" podUID="c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" containerName="maas-api" containerID="cri-o://4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d" gracePeriod=30 Oct 30 02:46:42.546061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.545982 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:46:42.635776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.635731 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-kube-api-access-lpd9n\" (UniqueName: \"kubernetes.io/projected/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-kube-api-access-lpd9n\") pod \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " Oct 30 02:46:42.636003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.635789 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-maas-api-tls\") pod \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " Oct 30 02:46:42.636003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.635828 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-metrics-certs\") pod \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " Oct 30 02:46:42.636003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.635933 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-openshift-service-ca\") pod \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\" (UID: \"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d\") " Oct 30 02:46:42.636367 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.636335 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-openshift-service-ca" pod "c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" (UID: "c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:42.638044 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.638016 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-maas-api-tls" pod "c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" (UID: "c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:42.638175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.638046 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-kube-api-access-lpd9n" pod "c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" (UID: "c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d"). InnerVolumeSpecName "kube-api-access-lpd9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:42.638175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.638046 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-metrics-certs" pod "c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" (UID: "c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:42.660201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.660152 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" podUID="c5a23052-76c4-4b5f-9e92-95aeb0d322e7" containerName="payload-processing" containerID="cri-o://d8333b16222fadb547455db18b1b76e0f0808a922bb88c1736b9f75ade3522dd" gracePeriod=25 Oct 30 02:46:42.696184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.696123 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" podUID="d6d4aa6e-66fa-4c8f-b1d0-916477867b42" containerName="payload-pre-processing" containerID="cri-o://fbd0e71f50b7b48f072a2e41adb03cc30e325c6de0d1e230e734392ca4ac8850" gracePeriod=25 Oct 30 02:46:42.738597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.738505 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:42.738597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.738590 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpd9n\" (UniqueName: \"kubernetes.io/projected/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-kube-api-access-lpd9n\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:42.738887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.738607 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:42.738887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.738622 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:42.793594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.793551 5035 generic.go:356] "Generic (PLEG): container finished" podID="c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" containerID="4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d" exitCode=0 Oct 30 02:46:42.793818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.793602 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" Oct 30 02:46:42.793818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.793633 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" event={"ID":"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d","Type":"ContainerDied","Data":"4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d"} Oct 30 02:46:42.793818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.793691 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5" event={"ID":"c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d","Type":"ContainerDied","Data":"f8aefff9ae8d5fe46c4d7315a4c0836a4cf1c1dec08d95d0d0050ac84c75eebb"} Oct 30 02:46:42.793818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.793719 5035 scope.go:122] "RemoveContainer" containerID="4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d" Oct 30 02:46:42.804660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.804629 5035 scope.go:122] "RemoveContainer" containerID="4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d" Oct 30 02:46:42.804964 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:42.804943 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d\": container with ID starting with 4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d not found: ID does not exist" containerID="4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d" Oct 30 02:46:42.805010 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.804976 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d"} err="failed to get container status \"4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d\": rpc error: code = NotFound desc = could not find container \"4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d\": container with ID starting with 4284eb6bfd9e3ed924007c6e72d20cc68adf41d6a45ccf10a0c87524ff084e4d not found: ID does not exist" Oct 30 02:46:42.845324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.845280 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5"] Oct 30 02:46:42.852337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:42.852304 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0fdc4e-66f4b468b-dxhq5"] Oct 30 02:46:43.219762 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.219727 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:46:43.345945 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.345842 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-data\") pod \"ea133939-45a3-47f2-886b-62ebad16f9a3\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " Oct 30 02:46:43.345945 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.345899 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-token\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-token\") pod \"ea133939-45a3-47f2-886b-62ebad16f9a3\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " Oct 30 02:46:43.346565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.345977 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-podinfo\") pod \"ea133939-45a3-47f2-886b-62ebad16f9a3\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " Oct 30 02:46:43.346565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346096 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-envoy\") pod \"ea133939-45a3-47f2-886b-62ebad16f9a3\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " Oct 30 02:46:43.346565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346165 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-kube-api-access-ln97n\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-kube-api-access-ln97n\") pod \"ea133939-45a3-47f2-886b-62ebad16f9a3\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " Oct 30 02:46:43.346565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346191 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-data" pod "ea133939-45a3-47f2-886b-62ebad16f9a3" (UID: "ea133939-45a3-47f2-886b-62ebad16f9a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.346565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346251 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-socket\") pod \"ea133939-45a3-47f2-886b-62ebad16f9a3\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " Oct 30 02:46:43.346565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346343 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-credential-socket\") pod \"ea133939-45a3-47f2-886b-62ebad16f9a3\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " Oct 30 02:46:43.346565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346474 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-certs\") pod \"ea133939-45a3-47f2-886b-62ebad16f9a3\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " Oct 30 02:46:43.346565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346504 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea133939-45a3-47f2-886b-62ebad16f9a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea133939-45a3-47f2-886b-62ebad16f9a3-istiod-ca-cert\") pod \"ea133939-45a3-47f2-886b-62ebad16f9a3\" (UID: \"ea133939-45a3-47f2-886b-62ebad16f9a3\") " Oct 30 02:46:43.346565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346541 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-socket" pod "ea133939-45a3-47f2-886b-62ebad16f9a3" (UID: "ea133939-45a3-47f2-886b-62ebad16f9a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.347110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346748 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-certs" pod "ea133939-45a3-47f2-886b-62ebad16f9a3" (UID: "ea133939-45a3-47f2-886b-62ebad16f9a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.347110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346688 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-credential-socket" pod "ea133939-45a3-47f2-886b-62ebad16f9a3" (UID: "ea133939-45a3-47f2-886b-62ebad16f9a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.347110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.346990 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea133939-45a3-47f2-886b-62ebad16f9a3-istiod-ca-cert" pod "ea133939-45a3-47f2-886b-62ebad16f9a3" (UID: "ea133939-45a3-47f2-886b-62ebad16f9a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:43.347296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.347122 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.347296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.347137 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.347296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.347147 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.347296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.347157 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea133939-45a3-47f2-886b-62ebad16f9a3-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.347296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.347167 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.348571 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.348540 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-token" pod "ea133939-45a3-47f2-886b-62ebad16f9a3" (UID: "ea133939-45a3-47f2-886b-62ebad16f9a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:43.348571 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.348563 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-kube-api-access-ln97n" pod "ea133939-45a3-47f2-886b-62ebad16f9a3" (UID: "ea133939-45a3-47f2-886b-62ebad16f9a3"). InnerVolumeSpecName "kube-api-access-ln97n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:43.348757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.348733 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-podinfo" pod "ea133939-45a3-47f2-886b-62ebad16f9a3" (UID: "ea133939-45a3-47f2-886b-62ebad16f9a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:46:43.348803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.348764 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-envoy" pod "ea133939-45a3-47f2-886b-62ebad16f9a3" (UID: "ea133939-45a3-47f2-886b-62ebad16f9a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.447953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.447905 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.447953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.447950 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.447953 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.447959 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea133939-45a3-47f2-886b-62ebad16f9a3-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.448222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.447970 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln97n\" (UniqueName: \"kubernetes.io/projected/ea133939-45a3-47f2-886b-62ebad16f9a3-kube-api-access-ln97n\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.643040 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.642949 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" path="/var/lib/kubelet/pods/c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d/volumes" Oct 30 02:46:43.802051 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.802008 5035 generic.go:356] "Generic (PLEG): container finished" podID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerID="49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894" exitCode=0 Oct 30 02:46:43.802272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.802063 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" Oct 30 02:46:43.802272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.802099 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" event={"ID":"ea133939-45a3-47f2-886b-62ebad16f9a3","Type":"ContainerDied","Data":"49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894"} Oct 30 02:46:43.802272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.802149 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck" event={"ID":"ea133939-45a3-47f2-886b-62ebad16f9a3","Type":"ContainerDied","Data":"0cb914d82615f6e210ded43ecda2a32f457c418be7da3b88697ef8d1a3cfefcd"} Oct 30 02:46:43.802272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.802171 5035 scope.go:122] "RemoveContainer" containerID="49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894" Oct 30 02:46:43.812519 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.812492 5035 scope.go:122] "RemoveContainer" containerID="49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894" Oct 30 02:46:43.812911 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:43.812888 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894\": container with ID starting with 49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894 not found: ID does not exist" containerID="49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894" Oct 30 02:46:43.812991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.812924 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894"} err="failed to get container status \"49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894\": rpc error: code = NotFound desc = could not find container \"49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894\": container with ID starting with 49718ab09b9790ec8cd0780e6d2efadaa3546784bca7ff926ce312bb36cb3894 not found: ID does not exist" Oct 30 02:46:43.854702 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.854664 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck"] Oct 30 02:46:43.873642 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:43.869700 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-rhvck"] Oct 30 02:46:45.643390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.643350 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" path="/var/lib/kubelet/pods/ea133939-45a3-47f2-886b-62ebad16f9a3/volumes" Oct 30 02:46:45.820429 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.820381 5035 generic.go:356] "Generic (PLEG): container finished" podID="ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" containerID="bec2352d71cf6d8b40f480c2369fa6cc62342d49053711cf6e4147cf13ed0209" exitCode=0 Oct 30 02:46:45.820622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.820515 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" event={"ID":"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765","Type":"ContainerDied","Data":"bec2352d71cf6d8b40f480c2369fa6cc62342d49053711cf6e4147cf13ed0209"} Oct 30 02:46:45.820622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.820588 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" event={"ID":"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765","Type":"ContainerDied","Data":"6469b597719b68ca2453a66436927737d0a3a796e54ddc3634578ff1b0b7d603"} Oct 30 02:46:45.820740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.820629 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6469b597719b68ca2453a66436927737d0a3a796e54ddc3634578ff1b0b7d603" Oct 30 02:46:45.822773 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.822735 5035 generic.go:356] "Generic (PLEG): container finished" podID="f4b57914-820a-470a-bb14-34c53a03840a" containerID="6f741b6659090c153a51a1480df61c017284033e30497216babb87f6b53e6c43" exitCode=0 Oct 30 02:46:45.822970 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.822822 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" event={"ID":"f4b57914-820a-470a-bb14-34c53a03840a","Type":"ContainerDied","Data":"6f741b6659090c153a51a1480df61c017284033e30497216babb87f6b53e6c43"} Oct 30 02:46:45.826013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.825978 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:46:45.863092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.863061 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:46:45.981392 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.981351 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b57914-820a-470a-bb14-34c53a03840a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b57914-820a-470a-bb14-34c53a03840a-plugins-config-volume\") pod \"f4b57914-820a-470a-bb14-34c53a03840a\" (UID: \"f4b57914-820a-470a-bb14-34c53a03840a\") " Oct 30 02:46:45.981657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.981473 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b57914-820a-470a-bb14-34c53a03840a-kube-api-access-jd87r\" (UniqueName: \"kubernetes.io/projected/f4b57914-820a-470a-bb14-34c53a03840a-kube-api-access-jd87r\") pod \"f4b57914-820a-470a-bb14-34c53a03840a\" (UID: \"f4b57914-820a-470a-bb14-34c53a03840a\") " Oct 30 02:46:45.981657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.981514 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-plugins-config-volume\") pod \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\" (UID: \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\") " Oct 30 02:46:45.981657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.981632 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-kube-api-access-8hktb\" (UniqueName: \"kubernetes.io/projected/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-kube-api-access-8hktb\") pod \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\" (UID: \"ab68786d-fb38-4ba1-bbaf-a5bbfcd96765\") " Oct 30 02:46:45.981966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.981934 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b57914-820a-470a-bb14-34c53a03840a-plugins-config-volume" pod "f4b57914-820a-470a-bb14-34c53a03840a" (UID: "f4b57914-820a-470a-bb14-34c53a03840a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:45.982022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.981993 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-plugins-config-volume" pod "ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" (UID: "ab68786d-fb38-4ba1-bbaf-a5bbfcd96765"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:45.982155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.982137 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:45.982198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.982162 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b57914-820a-470a-bb14-34c53a03840a-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:45.983826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.983800 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-kube-api-access-8hktb" pod "ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" (UID: "ab68786d-fb38-4ba1-bbaf-a5bbfcd96765"). InnerVolumeSpecName "kube-api-access-8hktb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:45.983915 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:45.983893 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b57914-820a-470a-bb14-34c53a03840a-kube-api-access-jd87r" pod "f4b57914-820a-470a-bb14-34c53a03840a" (UID: "f4b57914-820a-470a-bb14-34c53a03840a"). InnerVolumeSpecName "kube-api-access-jd87r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:46.083734 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.083640 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd87r\" (UniqueName: \"kubernetes.io/projected/f4b57914-820a-470a-bb14-34c53a03840a-kube-api-access-jd87r\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:46.083734 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.083675 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hktb\" (UniqueName: \"kubernetes.io/projected/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765-kube-api-access-8hktb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:46.830864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.830711 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" Oct 30 02:46:46.830864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.830711 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz" event={"ID":"f4b57914-820a-470a-bb14-34c53a03840a","Type":"ContainerDied","Data":"3668c389b4b185a9aa3e7ab17b745ed3452e4b8889b358e8d4a582129a109605"} Oct 30 02:46:46.831622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.830888 5035 scope.go:122] "RemoveContainer" containerID="6f741b6659090c153a51a1480df61c017284033e30497216babb87f6b53e6c43" Oct 30 02:46:46.831622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.831161 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5" Oct 30 02:46:46.897743 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.897703 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz"] Oct 30 02:46:46.909655 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.909606 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-584ec5-845b8d8d95-lr8hz"] Oct 30 02:46:46.934262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.934225 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5"] Oct 30 02:46:46.940622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:46.940588 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-584ec5-5bdd9dcf6d-bjvv5"] Oct 30 02:46:47.629987 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:47.629947 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:46:47.643077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:47.643041 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" path="/var/lib/kubelet/pods/ab68786d-fb38-4ba1-bbaf-a5bbfcd96765/volumes" Oct 30 02:46:47.643476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:47.643463 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b57914-820a-470a-bb14-34c53a03840a" path="/var/lib/kubelet/pods/f4b57914-820a-470a-bb14-34c53a03840a/volumes" Oct 30 02:46:49.911468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:49.911428 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg"] Oct 30 02:46:49.912058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:49.911706 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" podUID="b81cd9b8-f172-4579-85fe-b447b1187777" containerName="maas-api" containerID="cri-o://4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd" gracePeriod=30 Oct 30 02:46:50.024754 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.024719 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" podUID="d6d4aa6e-66fa-4c8f-b1d0-916477867b42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:50.193371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.193337 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:46:50.339286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.339239 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-metrics-certs\") pod \"b81cd9b8-f172-4579-85fe-b447b1187777\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " Oct 30 02:46:50.339286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.339295 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b81cd9b8-f172-4579-85fe-b447b1187777-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cd9b8-f172-4579-85fe-b447b1187777-openshift-service-ca\") pod \"b81cd9b8-f172-4579-85fe-b447b1187777\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " Oct 30 02:46:50.339523 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.339414 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls\") pod \"b81cd9b8-f172-4579-85fe-b447b1187777\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " Oct 30 02:46:50.339523 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.339480 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81cd9b8-f172-4579-85fe-b447b1187777-kube-api-access-4nn5c\" (UniqueName: \"kubernetes.io/projected/b81cd9b8-f172-4579-85fe-b447b1187777-kube-api-access-4nn5c\") pod \"b81cd9b8-f172-4579-85fe-b447b1187777\" (UID: \"b81cd9b8-f172-4579-85fe-b447b1187777\") " Oct 30 02:46:50.339778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.339748 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81cd9b8-f172-4579-85fe-b447b1187777-openshift-service-ca" pod "b81cd9b8-f172-4579-85fe-b447b1187777" (UID: "b81cd9b8-f172-4579-85fe-b447b1187777"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:50.340042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.340015 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cd9b8-f172-4579-85fe-b447b1187777-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:50.341821 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.341787 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81cd9b8-f172-4579-85fe-b447b1187777-kube-api-access-4nn5c" pod "b81cd9b8-f172-4579-85fe-b447b1187777" (UID: "b81cd9b8-f172-4579-85fe-b447b1187777"). InnerVolumeSpecName "kube-api-access-4nn5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:50.341821 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.341807 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls" pod "b81cd9b8-f172-4579-85fe-b447b1187777" (UID: "b81cd9b8-f172-4579-85fe-b447b1187777"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:50.341955 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.341835 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-metrics-certs" pod "b81cd9b8-f172-4579-85fe-b447b1187777" (UID: "b81cd9b8-f172-4579-85fe-b447b1187777"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:50.441468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.441362 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:50.441468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.441407 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nn5c\" (UniqueName: \"kubernetes.io/projected/b81cd9b8-f172-4579-85fe-b447b1187777-kube-api-access-4nn5c\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:50.441468 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.441421 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b81cd9b8-f172-4579-85fe-b447b1187777-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:46:50.857806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.857770 5035 generic.go:356] "Generic (PLEG): container finished" podID="b81cd9b8-f172-4579-85fe-b447b1187777" containerID="4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd" exitCode=0 Oct 30 02:46:50.858013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.857819 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" Oct 30 02:46:50.858013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.857843 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" event={"ID":"b81cd9b8-f172-4579-85fe-b447b1187777","Type":"ContainerDied","Data":"4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd"} Oct 30 02:46:50.858013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.857887 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg" event={"ID":"b81cd9b8-f172-4579-85fe-b447b1187777","Type":"ContainerDied","Data":"9c0afc0556842c593791d64222aab174ee3772854b9f0be3a7174d012255101a"} Oct 30 02:46:50.858013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.857908 5035 scope.go:122] "RemoveContainer" containerID="4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd" Oct 30 02:46:50.870021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.869998 5035 scope.go:122] "RemoveContainer" containerID="4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd" Oct 30 02:46:50.870358 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:46:50.870335 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd\": container with ID starting with 4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd not found: ID does not exist" containerID="4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd" Oct 30 02:46:50.870429 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.870369 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd"} err="failed to get container status \"4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd\": rpc error: code = NotFound desc = could not find container \"4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd\": container with ID starting with 4b780089bfe94b720a4bc8a0c6c365fec5f128722e48d89ebc92c4e22d5e75cd not found: ID does not exist" Oct 30 02:46:50.908881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.908844 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg"] Oct 30 02:46:50.920826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:50.920790 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-24fb97-6b789789bb-br2tg"] Oct 30 02:46:51.032850 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:51.032803 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" podUID="c5a23052-76c4-4b5f-9e92-95aeb0d322e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:46:51.643193 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:51.643156 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b81cd9b8-f172-4579-85fe-b447b1187777" path="/var/lib/kubelet/pods/b81cd9b8-f172-4579-85fe-b447b1187777/volumes" Oct 30 02:46:54.965937 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:54.965893 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd"] Oct 30 02:46:54.966462 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:54.966160 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" containerID="cri-o://ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099" gracePeriod=30 Oct 30 02:46:58.078372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:58.078316 5035 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:46:58.078896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:46:58.078387 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:47:00.024622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.024575 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" podUID="d6d4aa6e-66fa-4c8f-b1d0-916477867b42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:00.254234 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.254154 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:47:00.389345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.389271 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-kube-api-access-5s7g4\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-kube-api-access-5s7g4\") pod \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " Oct 30 02:47:00.389766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.389361 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-data\") pod \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " Oct 30 02:47:00.389766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.389442 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-certs\") pod \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " Oct 30 02:47:00.389766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.389740 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istiod-ca-cert\") pod \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " Oct 30 02:47:00.390062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.389792 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-token\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-token\") pod \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " Oct 30 02:47:00.390062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.389857 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-envoy\") pod \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " Oct 30 02:47:00.390062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.389856 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-data" pod "14f7a1fb-dc50-4f51-ab08-e745a1a1703b" (UID: "14f7a1fb-dc50-4f51-ab08-e745a1a1703b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:00.390062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.389933 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-certs" pod "14f7a1fb-dc50-4f51-ab08-e745a1a1703b" (UID: "14f7a1fb-dc50-4f51-ab08-e745a1a1703b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:00.390062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.389974 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-socket\") pod \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " Oct 30 02:47:00.390062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.390034 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-podinfo\") pod \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " Oct 30 02:47:00.390350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.390087 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-credential-socket\") pod \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\" (UID: \"14f7a1fb-dc50-4f51-ab08-e745a1a1703b\") " Oct 30 02:47:00.390350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.390285 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istiod-ca-cert" pod "14f7a1fb-dc50-4f51-ab08-e745a1a1703b" (UID: "14f7a1fb-dc50-4f51-ab08-e745a1a1703b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:00.390350 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.390296 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-socket" pod "14f7a1fb-dc50-4f51-ab08-e745a1a1703b" (UID: "14f7a1fb-dc50-4f51-ab08-e745a1a1703b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:00.390606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.390576 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-credential-socket" pod "14f7a1fb-dc50-4f51-ab08-e745a1a1703b" (UID: "14f7a1fb-dc50-4f51-ab08-e745a1a1703b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:00.390795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.390701 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.390892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.390811 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.390892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.390832 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.390892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.390864 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.392639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.392592 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-token" pod "14f7a1fb-dc50-4f51-ab08-e745a1a1703b" (UID: "14f7a1fb-dc50-4f51-ab08-e745a1a1703b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:00.392639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.392617 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-kube-api-access-5s7g4" pod "14f7a1fb-dc50-4f51-ab08-e745a1a1703b" (UID: "14f7a1fb-dc50-4f51-ab08-e745a1a1703b"). InnerVolumeSpecName "kube-api-access-5s7g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:00.393063 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.393035 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-podinfo" pod "14f7a1fb-dc50-4f51-ab08-e745a1a1703b" (UID: "14f7a1fb-dc50-4f51-ab08-e745a1a1703b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:47:00.393234 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.393201 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-envoy" pod "14f7a1fb-dc50-4f51-ab08-e745a1a1703b" (UID: "14f7a1fb-dc50-4f51-ab08-e745a1a1703b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:00.459499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.459418 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn"] Oct 30 02:47:00.466384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.466299 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946"] Oct 30 02:47:00.476992 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.476845 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z"] Oct 30 02:47:00.493761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.493679 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.493761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.493752 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.493761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.493766 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.493761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.493785 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.493761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.493811 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s7g4\" (UniqueName: \"kubernetes.io/projected/14f7a1fb-dc50-4f51-ab08-e745a1a1703b-kube-api-access-5s7g4\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.926863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.926820 5035 generic.go:356] "Generic (PLEG): container finished" podID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerID="ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099" exitCode=0 Oct 30 02:47:00.927038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.926875 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" Oct 30 02:47:00.927038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.926898 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" event={"ID":"14f7a1fb-dc50-4f51-ab08-e745a1a1703b","Type":"ContainerDied","Data":"ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099"} Oct 30 02:47:00.927038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.926959 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd" event={"ID":"14f7a1fb-dc50-4f51-ab08-e745a1a1703b","Type":"ContainerDied","Data":"109373491cfe6c612bf9426949af441929bc8d033b7eb63fb82d247983fc6c69"} Oct 30 02:47:00.927038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.926987 5035 scope.go:122] "RemoveContainer" containerID="ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099" Oct 30 02:47:00.938221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.938197 5035 scope.go:122] "RemoveContainer" containerID="ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b" Oct 30 02:47:00.948442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.948416 5035 scope.go:122] "RemoveContainer" containerID="ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099" Oct 30 02:47:00.948786 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:00.948763 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099\": container with ID starting with ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099 not found: ID does not exist" containerID="ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099" Oct 30 02:47:00.948878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.948797 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099"} err="failed to get container status \"ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099\": rpc error: code = NotFound desc = could not find container \"ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099\": container with ID starting with ab1140a7815542d6aedc1265ab9d41e73805a64fb09ba252904b2b8c72ad1099 not found: ID does not exist" Oct 30 02:47:00.948878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.948815 5035 scope.go:122] "RemoveContainer" containerID="ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b" Oct 30 02:47:00.949096 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:00.949073 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b\": container with ID starting with ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b not found: ID does not exist" containerID="ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b" Oct 30 02:47:00.949155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:00.949107 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b"} err="failed to get container status \"ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b\": rpc error: code = NotFound desc = could not find container \"ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b\": container with ID starting with ef98ff6680feaa528f65cd173321824d6942ebc5c485c89d4c4d5ea21a0c7b8b not found: ID does not exist" Oct 30 02:47:01.005264 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:01.005227 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd"] Oct 30 02:47:01.017264 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:01.017226 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01a9cd-openshift-default-8468779c9c-n7zbd"] Oct 30 02:47:01.033271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:01.033216 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" podUID="c5a23052-76c4-4b5f-9e92-95aeb0d322e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:01.643011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:01.642979 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" path="/var/lib/kubelet/pods/14f7a1fb-dc50-4f51-ab08-e745a1a1703b/volumes" Oct 30 02:47:02.944499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:02.944462 5035 generic.go:356] "Generic (PLEG): container finished" podID="c5a23052-76c4-4b5f-9e92-95aeb0d322e7" containerID="d8333b16222fadb547455db18b1b76e0f0808a922bb88c1736b9f75ade3522dd" exitCode=0 Oct 30 02:47:02.945073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:02.944994 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" event={"ID":"c5a23052-76c4-4b5f-9e92-95aeb0d322e7","Type":"ContainerDied","Data":"d8333b16222fadb547455db18b1b76e0f0808a922bb88c1736b9f75ade3522dd"} Oct 30 02:47:02.954960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:02.954812 5035 generic.go:356] "Generic (PLEG): container finished" podID="d6d4aa6e-66fa-4c8f-b1d0-916477867b42" containerID="fbd0e71f50b7b48f072a2e41adb03cc30e325c6de0d1e230e734392ca4ac8850" exitCode=0 Oct 30 02:47:02.954960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:02.954943 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" event={"ID":"d6d4aa6e-66fa-4c8f-b1d0-916477867b42","Type":"ContainerDied","Data":"fbd0e71f50b7b48f072a2e41adb03cc30e325c6de0d1e230e734392ca4ac8850"} Oct 30 02:47:02.977155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:02.977122 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:47:03.017174 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.017140 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:47:03.128335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.128298 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-plugins-config-volume\") pod \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\" (UID: \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\") " Oct 30 02:47:03.128577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.128442 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-kube-api-access-55bhk\" (UniqueName: \"kubernetes.io/projected/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-kube-api-access-55bhk\") pod \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\" (UID: \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\") " Oct 30 02:47:03.128577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.128495 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-plugins-config-volume\") pod \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\" (UID: \"c5a23052-76c4-4b5f-9e92-95aeb0d322e7\") " Oct 30 02:47:03.128577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.128561 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-kube-api-access-7qbq9\" (UniqueName: \"kubernetes.io/projected/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-kube-api-access-7qbq9\") pod \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\" (UID: \"d6d4aa6e-66fa-4c8f-b1d0-916477867b42\") " Oct 30 02:47:03.128860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.128831 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-plugins-config-volume" pod "d6d4aa6e-66fa-4c8f-b1d0-916477867b42" (UID: "d6d4aa6e-66fa-4c8f-b1d0-916477867b42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:03.128933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.128863 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-plugins-config-volume" pod "c5a23052-76c4-4b5f-9e92-95aeb0d322e7" (UID: "c5a23052-76c4-4b5f-9e92-95aeb0d322e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:03.129019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.129002 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:03.129066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.129022 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:03.130809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.130789 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-kube-api-access-7qbq9" pod "d6d4aa6e-66fa-4c8f-b1d0-916477867b42" (UID: "d6d4aa6e-66fa-4c8f-b1d0-916477867b42"). InnerVolumeSpecName "kube-api-access-7qbq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:03.130886 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.130807 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-kube-api-access-55bhk" pod "c5a23052-76c4-4b5f-9e92-95aeb0d322e7" (UID: "c5a23052-76c4-4b5f-9e92-95aeb0d322e7"). InnerVolumeSpecName "kube-api-access-55bhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:03.230169 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.230131 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55bhk\" (UniqueName: \"kubernetes.io/projected/c5a23052-76c4-4b5f-9e92-95aeb0d322e7-kube-api-access-55bhk\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:03.230169 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.230165 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qbq9\" (UniqueName: \"kubernetes.io/projected/d6d4aa6e-66fa-4c8f-b1d0-916477867b42-kube-api-access-7qbq9\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:03.479088 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.479009 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9"] Oct 30 02:47:03.507686 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.507593 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj"] Oct 30 02:47:03.964483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.964451 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" Oct 30 02:47:03.965057 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.964452 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp" event={"ID":"d6d4aa6e-66fa-4c8f-b1d0-916477867b42","Type":"ContainerDied","Data":"9b9f913444231e8eb94d670424400095b1d7aad42cdf3aecce1559ee2ddd7cc4"} Oct 30 02:47:03.965057 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.964669 5035 scope.go:122] "RemoveContainer" containerID="fbd0e71f50b7b48f072a2e41adb03cc30e325c6de0d1e230e734392ca4ac8850" Oct 30 02:47:03.966669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.966641 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" event={"ID":"c5a23052-76c4-4b5f-9e92-95aeb0d322e7","Type":"ContainerDied","Data":"cd6036ca7a0a53fa718f59058235854c583f7fd2ee89cd482f93ac6aa98d1582"} Oct 30 02:47:03.966669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.966659 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9" Oct 30 02:47:03.975565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:03.975540 5035 scope.go:122] "RemoveContainer" containerID="d8333b16222fadb547455db18b1b76e0f0808a922bb88c1736b9f75ade3522dd" Oct 30 02:47:04.036464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:04.036416 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9"] Oct 30 02:47:04.048345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:04.048307 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-24fb97-f5bb9c87d-xwxd9"] Oct 30 02:47:04.089855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:04.089815 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp"] Oct 30 02:47:04.094298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:04.094265 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-24fb97-7b6cbd5699-nc5tp"] Oct 30 02:47:05.644342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:05.644307 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a23052-76c4-4b5f-9e92-95aeb0d322e7" path="/var/lib/kubelet/pods/c5a23052-76c4-4b5f-9e92-95aeb0d322e7/volumes" Oct 30 02:47:05.644793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:05.644749 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d4aa6e-66fa-4c8f-b1d0-916477867b42" path="/var/lib/kubelet/pods/d6d4aa6e-66fa-4c8f-b1d0-916477867b42/volumes" Oct 30 02:47:05.926809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:05.926698 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms"] Oct 30 02:47:05.927017 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:05.926973 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" podUID="92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" containerName="maas-api" containerID="cri-o://891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93" gracePeriod=30 Oct 30 02:47:06.264714 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.264687 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:47:06.379572 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.379514 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-metrics-certs\") pod \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " Oct 30 02:47:06.380055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.379612 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls\") pod \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " Oct 30 02:47:06.380055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.379691 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-openshift-service-ca\") pod \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " Oct 30 02:47:06.380055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.379748 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-kube-api-access-h2qf7\" (UniqueName: \"kubernetes.io/projected/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-kube-api-access-h2qf7\") pod \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\" (UID: \"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2\") " Oct 30 02:47:06.380801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.380693 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-openshift-service-ca" pod "92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" (UID: "92c1cf7d-f6ca-43b7-aff3-99b29fd127c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:06.381248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.381196 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:06.385147 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.385015 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls" pod "92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" (UID: "92c1cf7d-f6ca-43b7-aff3-99b29fd127c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:06.385656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.385129 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-kube-api-access-h2qf7" pod "92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" (UID: "92c1cf7d-f6ca-43b7-aff3-99b29fd127c2"). InnerVolumeSpecName "kube-api-access-h2qf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:06.385656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.385048 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-metrics-certs" pod "92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" (UID: "92c1cf7d-f6ca-43b7-aff3-99b29fd127c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:06.482263 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.482162 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:06.482263 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.482204 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:06.482263 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.482213 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2qf7\" (UniqueName: \"kubernetes.io/projected/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2-kube-api-access-h2qf7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:06.991457 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.991417 5035 generic.go:356] "Generic (PLEG): container finished" podID="92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" containerID="891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93" exitCode=0 Oct 30 02:47:06.991968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.991468 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" Oct 30 02:47:06.991968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.991481 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" event={"ID":"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2","Type":"ContainerDied","Data":"891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93"} Oct 30 02:47:06.991968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.991523 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms" event={"ID":"92c1cf7d-f6ca-43b7-aff3-99b29fd127c2","Type":"ContainerDied","Data":"dedb443ad837048bfdcae042ecc640936c3134a6deb1565570b21aba3f8a835f"} Oct 30 02:47:06.991968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:06.991567 5035 scope.go:122] "RemoveContainer" containerID="891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93" Oct 30 02:47:07.005805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.005780 5035 scope.go:122] "RemoveContainer" containerID="891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93" Oct 30 02:47:07.006152 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:07.006127 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93\": container with ID starting with 891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93 not found: ID does not exist" containerID="891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93" Oct 30 02:47:07.006222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.006161 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93"} err="failed to get container status \"891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93\": rpc error: code = NotFound desc = could not find container \"891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93\": container with ID starting with 891fdf58b653e5f8cdf2dbbb5bf80bc2baa7f203811b58edf3590f969d1ecf93 not found: ID does not exist" Oct 30 02:47:07.043178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.043143 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms"] Oct 30 02:47:07.055300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.055260 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-594253-78456f6bc7-5rrms"] Oct 30 02:47:07.625989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.625937 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz"] Oct 30 02:47:07.648200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.648156 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" path="/var/lib/kubelet/pods/92c1cf7d-f6ca-43b7-aff3-99b29fd127c2/volumes" Oct 30 02:47:07.656091 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.656054 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6"] Oct 30 02:47:07.990470 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.990424 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c"] Oct 30 02:47:07.991035 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991014 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c5a23052-76c4-4b5f-9e92-95aeb0d322e7" containerName="payload-processing" Oct 30 02:47:07.991105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991037 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a23052-76c4-4b5f-9e92-95aeb0d322e7" containerName="payload-processing" Oct 30 02:47:07.991105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991051 5035 cpu_manager.go:414] "containerMap: removing container" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerName="cleanup" Oct 30 02:47:07.991105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991057 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerName="cleanup" Oct 30 02:47:07.991105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991068 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f4b57914-820a-470a-bb14-34c53a03840a" containerName="payload-pre-processing" Oct 30 02:47:07.991105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991074 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b57914-820a-470a-bb14-34c53a03840a" containerName="payload-pre-processing" Oct 30 02:47:07.991105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991082 5035 cpu_manager.go:414] "containerMap: removing container" podUID="802465ef-c6e6-40eb-b083-c3fa61aed348" containerName="payload-pre-processing" Oct 30 02:47:07.991105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991087 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802465ef-c6e6-40eb-b083-c3fa61aed348" containerName="payload-pre-processing" Oct 30 02:47:07.991105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991101 5035 cpu_manager.go:414] "containerMap: removing container" podUID="5a90a801-f3b4-439b-b4b5-2a5ece592228" containerName="payload-processing" Oct 30 02:47:07.991105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991107 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a90a801-f3b4-439b-b4b5-2a5ece592228" containerName="payload-processing" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991119 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991124 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991132 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e7417f30-4ad3-47c8-835d-b0511309cac5" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991137 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7417f30-4ad3-47c8-835d-b0511309cac5" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991148 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" containerName="payload-processing" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991153 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" containerName="payload-processing" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991162 5035 cpu_manager.go:414] "containerMap: removing container" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991168 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991180 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991184 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991191 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991196 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991205 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b81cd9b8-f172-4579-85fe-b447b1187777" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991209 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81cd9b8-f172-4579-85fe-b447b1187777" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991217 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d6d4aa6e-66fa-4c8f-b1d0-916477867b42" containerName="payload-pre-processing" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991222 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d4aa6e-66fa-4c8f-b1d0-916477867b42" containerName="payload-pre-processing" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991232 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991236 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991243 5035 cpu_manager.go:414] "containerMap: removing container" podUID="952801bc-170d-499d-83e1-781126e7cda2" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991247 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952801bc-170d-499d-83e1-781126e7cda2" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991255 5035 cpu_manager.go:414] "containerMap: removing container" podUID="952801bc-170d-499d-83e1-781126e7cda2" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991260 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952801bc-170d-499d-83e1-781126e7cda2" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991270 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991275 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991281 5035 cpu_manager.go:414] "containerMap: removing container" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991286 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991294 5035 cpu_manager.go:414] "containerMap: removing container" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991298 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991306 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991310 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerName="cleanup" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991320 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991324 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991331 5035 cpu_manager.go:414] "containerMap: removing container" podUID="2bb4a780-b282-43f7-9173-d6bf26315614" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991337 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb4a780-b282-43f7-9173-d6bf26315614" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991344 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ea090139-df87-41c5-a7ca-6880c9ef8d8d" containerName="revoke-keys" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991349 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea090139-df87-41c5-a7ca-6880c9ef8d8d" containerName="revoke-keys" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991357 5035 cpu_manager.go:414] "containerMap: removing container" podUID="92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991361 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991369 5035 cpu_manager.go:414] "containerMap: removing container" podUID="053375c1-4f44-4ae1-9272-cbd76b067d5a" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991373 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053375c1-4f44-4ae1-9272-cbd76b067d5a" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991385 5035 cpu_manager.go:414] "containerMap: removing container" podUID="708b87c9-902e-4882-9b48-1ad131b19385" containerName="payload-pre-processing" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991390 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708b87c9-902e-4882-9b48-1ad131b19385" containerName="payload-pre-processing" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991401 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d20aac51-0f1a-4f75-81d3-33f010361b2e" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991406 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20aac51-0f1a-4f75-81d3-33f010361b2e" containerName="maas-api" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991412 5035 cpu_manager.go:414] "containerMap: removing container" podUID="2f369a52-a39c-490e-bf37-22bb44483960" containerName="main" Oct 30 02:47:07.991455 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991417 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f369a52-a39c-490e-bf37-22bb44483960" containerName="main" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991494 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81cd9b8-f172-4579-85fe-b447b1187777" containerName="maas-api" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991502 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d4aa6e-66fa-4c8f-b1d0-916477867b42" containerName="payload-pre-processing" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991510 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerName="cleanup" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991516 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="952801bc-170d-499d-83e1-781126e7cda2" containerName="cleanup" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991537 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="952801bc-170d-499d-83e1-781126e7cda2" containerName="cleanup" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991545 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerName="cleanup" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991552 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991559 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerName="cleanup" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991564 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerName="cleanup" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991571 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea133939-45a3-47f2-886b-62ebad16f9a3" containerName="istio-proxy" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991577 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb4a780-b282-43f7-9173-d6bf26315614" containerName="maas-api" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991582 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea090139-df87-41c5-a7ca-6880c9ef8d8d" containerName="revoke-keys" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991588 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c1cf7d-f6ca-43b7-aff3-99b29fd127c2" containerName="maas-api" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991592 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="053375c1-4f44-4ae1-9272-cbd76b067d5a" containerName="maas-api" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991599 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="708b87c9-902e-4882-9b48-1ad131b19385" containerName="payload-pre-processing" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991606 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20aac51-0f1a-4f75-81d3-33f010361b2e" containerName="maas-api" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991611 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f369a52-a39c-490e-bf37-22bb44483960" containerName="main" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991616 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerName="cleanup" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991621 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a23052-76c4-4b5f-9e92-95aeb0d322e7" containerName="payload-processing" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991627 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerName="cleanup" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991633 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b57914-820a-470a-bb14-34c53a03840a" containerName="payload-pre-processing" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991639 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="802465ef-c6e6-40eb-b083-c3fa61aed348" containerName="payload-pre-processing" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991646 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a90a801-f3b4-439b-b4b5-2a5ece592228" containerName="payload-processing" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991651 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7417f30-4ad3-47c8-835d-b0511309cac5" containerName="maas-api" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991657 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab68786d-fb38-4ba1-bbaf-a5bbfcd96765" containerName="payload-processing" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991662 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19dfbcf-ada7-4a8e-b41b-9432ebe58e3d" containerName="maas-api" Oct 30 02:47:07.993593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.991668 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerName="cleanup" Oct 30 02:47:07.996878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:07.996847 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.001142 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.001070 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-765f4b3e-6zr2c\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.001685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.001236 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76g85\" (UniqueName: \"kubernetes.io/projected/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-kube-api-access-76g85\") pod \"maas-api-revoke-sub-765f4b3e-6zr2c\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.001873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.001827 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-765f4b3e-6zr2c\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.015003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.013350 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:47:08.015003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.013350 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:47:08.015003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.013727 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:47:08.015003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.013972 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:47:08.022182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.022056 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c"] Oct 30 02:47:08.103459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.103410 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-765f4b3e-6zr2c\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.103714 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.103478 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-765f4b3e-6zr2c\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.103714 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.103513 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76g85\" (UniqueName: \"kubernetes.io/projected/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-kube-api-access-76g85\") pod \"maas-api-revoke-sub-765f4b3e-6zr2c\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.104318 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.104275 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-765f4b3e-6zr2c\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.106218 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.106194 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-765f4b3e-6zr2c\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.127703 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.127655 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76g85\" (UniqueName: \"kubernetes.io/projected/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-kube-api-access-76g85\") pod \"maas-api-revoke-sub-765f4b3e-6zr2c\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.331878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.331772 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:08.479499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.479440 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" podUID="02fd08d5-b5af-4b0c-9a55-e789cd75e036" containerName="payload-processing" containerID="cri-o://d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9" gracePeriod=25 Oct 30 02:47:08.508103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.508046 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" podUID="350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" containerName="payload-pre-processing" containerID="cri-o://3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541" gracePeriod=25 Oct 30 02:47:08.639036 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:08.639005 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c"] Oct 30 02:47:08.641337 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:47:08.641304 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd77c46c_19ee_4941_b3f7_4c7c3c2a47bb.slice/crio-787c977be25f8d16e1a7c2038f9484f36d01d54e55517367bb44536e3370f843 WatchSource:0}: Error finding container 787c977be25f8d16e1a7c2038f9484f36d01d54e55517367bb44536e3370f843: Status 404 returned error can't find the container with id 787c977be25f8d16e1a7c2038f9484f36d01d54e55517367bb44536e3370f843 Oct 30 02:47:09.012876 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:09.012829 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" event={"ID":"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb","Type":"ContainerStarted","Data":"472826932a5b254360637ef8de38f8ecd616873ace612b8936dfa19562e9b2ec"} Oct 30 02:47:09.013283 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:09.012889 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" event={"ID":"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb","Type":"ContainerStarted","Data":"787c977be25f8d16e1a7c2038f9484f36d01d54e55517367bb44536e3370f843"} Oct 30 02:47:09.057036 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:09.056966 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" podStartSLOduration=2.056948049 podStartE2EDuration="2.056948049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:47:09.055497544 +0000 UTC m=+4707.558429763" watchObservedRunningTime="2026-10-30 02:47:09.056948049 +0000 UTC m=+4707.559880270" Oct 30 02:47:09.867395 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:09.867288 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw"] Oct 30 02:47:10.020192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:10.020153 5035 generic.go:356] "Generic (PLEG): container finished" podID="cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb" containerID="472826932a5b254360637ef8de38f8ecd616873ace612b8936dfa19562e9b2ec" exitCode=0 Oct 30 02:47:10.020636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:10.020235 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" event={"ID":"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb","Type":"ContainerDied","Data":"472826932a5b254360637ef8de38f8ecd616873ace612b8936dfa19562e9b2ec"} Oct 30 02:47:11.187965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.187935 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:11.240487 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.240448 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-xks-ca\") pod \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " Oct 30 02:47:11.240487 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.240492 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-kube-api-access-76g85\" (UniqueName: \"kubernetes.io/projected/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-kube-api-access-76g85\") pod \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " Oct 30 02:47:11.240793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.240582 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-openshift-ca\") pod \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\" (UID: \"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb\") " Oct 30 02:47:11.241613 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.241580 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-openshift-ca" pod "cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb" (UID: "cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:11.243108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.243077 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-xks-ca" pod "cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb" (UID: "cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:11.243888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.243860 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-kube-api-access-76g85" pod "cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb" (UID: "cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb"). InnerVolumeSpecName "kube-api-access-76g85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:11.342669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.342513 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:11.342669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.342601 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76g85\" (UniqueName: \"kubernetes.io/projected/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-kube-api-access-76g85\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:11.342669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:11.342616 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:12.038160 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.038126 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" Oct 30 02:47:12.038365 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.038161 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c" event={"ID":"cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb","Type":"ContainerDied","Data":"787c977be25f8d16e1a7c2038f9484f36d01d54e55517367bb44536e3370f843"} Oct 30 02:47:12.038365 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.038210 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="787c977be25f8d16e1a7c2038f9484f36d01d54e55517367bb44536e3370f843" Oct 30 02:47:12.626989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.626939 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" podUID="13653c11-c510-447b-90fb-2d959bee34d0" containerName="payload-processing" containerID="cri-o://e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787" gracePeriod=25 Oct 30 02:47:12.656931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.656885 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" podUID="6f27a327-a17e-45e6-80fe-88d732edb488" containerName="payload-pre-processing" containerID="cri-o://16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7" gracePeriod=25 Oct 30 02:47:12.925416 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.925370 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 02:47:12.964346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.964297 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13653c11-c510-447b-90fb-2d959bee34d0-kube-api-access-p72l7\" (UniqueName: \"kubernetes.io/projected/13653c11-c510-447b-90fb-2d959bee34d0-kube-api-access-p72l7\") pod \"13653c11-c510-447b-90fb-2d959bee34d0\" (UID: \"13653c11-c510-447b-90fb-2d959bee34d0\") " Oct 30 02:47:12.964640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.964356 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13653c11-c510-447b-90fb-2d959bee34d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13653c11-c510-447b-90fb-2d959bee34d0-plugins-config-volume\") pod \"13653c11-c510-447b-90fb-2d959bee34d0\" (UID: \"13653c11-c510-447b-90fb-2d959bee34d0\") " Oct 30 02:47:12.965091 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.965045 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13653c11-c510-447b-90fb-2d959bee34d0-plugins-config-volume" pod "13653c11-c510-447b-90fb-2d959bee34d0" (UID: "13653c11-c510-447b-90fb-2d959bee34d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:12.967483 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.967436 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13653c11-c510-447b-90fb-2d959bee34d0-kube-api-access-p72l7" pod "13653c11-c510-447b-90fb-2d959bee34d0" (UID: "13653c11-c510-447b-90fb-2d959bee34d0"). InnerVolumeSpecName "kube-api-access-p72l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:12.986930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:12.986882 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 02:47:13.066016 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.065973 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f27a327-a17e-45e6-80fe-88d732edb488-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f27a327-a17e-45e6-80fe-88d732edb488-plugins-config-volume\") pod \"6f27a327-a17e-45e6-80fe-88d732edb488\" (UID: \"6f27a327-a17e-45e6-80fe-88d732edb488\") " Oct 30 02:47:13.066241 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.066062 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f27a327-a17e-45e6-80fe-88d732edb488-kube-api-access-4ht4f\" (UniqueName: \"kubernetes.io/projected/6f27a327-a17e-45e6-80fe-88d732edb488-kube-api-access-4ht4f\") pod \"6f27a327-a17e-45e6-80fe-88d732edb488\" (UID: \"6f27a327-a17e-45e6-80fe-88d732edb488\") " Oct 30 02:47:13.066508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.066479 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f27a327-a17e-45e6-80fe-88d732edb488-plugins-config-volume" pod "6f27a327-a17e-45e6-80fe-88d732edb488" (UID: "6f27a327-a17e-45e6-80fe-88d732edb488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:13.067014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.066991 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p72l7\" (UniqueName: \"kubernetes.io/projected/13653c11-c510-447b-90fb-2d959bee34d0-kube-api-access-p72l7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:13.067101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.067018 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13653c11-c510-447b-90fb-2d959bee34d0-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:13.067101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.067032 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f27a327-a17e-45e6-80fe-88d732edb488-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:13.069126 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.069095 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f27a327-a17e-45e6-80fe-88d732edb488-kube-api-access-4ht4f" pod "6f27a327-a17e-45e6-80fe-88d732edb488" (UID: "6f27a327-a17e-45e6-80fe-88d732edb488"). InnerVolumeSpecName "kube-api-access-4ht4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:13.069963 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.069935 5035 generic.go:356] "Generic (PLEG): container finished" podID="6f27a327-a17e-45e6-80fe-88d732edb488" containerID="16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7" exitCode=0 Oct 30 02:47:13.070063 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.070008 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" Oct 30 02:47:13.070063 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.070015 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" event={"ID":"6f27a327-a17e-45e6-80fe-88d732edb488","Type":"ContainerDied","Data":"16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7"} Oct 30 02:47:13.070174 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.070106 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6" event={"ID":"6f27a327-a17e-45e6-80fe-88d732edb488","Type":"ContainerDied","Data":"0a440c8726ff3481e4de8e0385eed75934bee1940f0a09af018024645e317f01"} Oct 30 02:47:13.070174 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.070125 5035 scope.go:122] "RemoveContainer" containerID="16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7" Oct 30 02:47:13.071719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.071690 5035 generic.go:356] "Generic (PLEG): container finished" podID="13653c11-c510-447b-90fb-2d959bee34d0" containerID="e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787" exitCode=0 Oct 30 02:47:13.071826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.071744 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" event={"ID":"13653c11-c510-447b-90fb-2d959bee34d0","Type":"ContainerDied","Data":"e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787"} Oct 30 02:47:13.071826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.071785 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" event={"ID":"13653c11-c510-447b-90fb-2d959bee34d0","Type":"ContainerDied","Data":"091e209390eb7448e482d8e89cb33fb4cd54a53989917863ab47c6c892c00ceb"} Oct 30 02:47:13.071968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.071950 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz" Oct 30 02:47:13.080795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.080766 5035 scope.go:122] "RemoveContainer" containerID="16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7" Oct 30 02:47:13.081157 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:13.081128 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7\": container with ID starting with 16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7 not found: ID does not exist" containerID="16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7" Oct 30 02:47:13.081247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.081158 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7"} err="failed to get container status \"16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7\": rpc error: code = NotFound desc = could not find container \"16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7\": container with ID starting with 16565eea94fd024253b5ed5879decb993c56c217f3e536e2320ae959a48c92a7 not found: ID does not exist" Oct 30 02:47:13.081247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.081180 5035 scope.go:122] "RemoveContainer" containerID="e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787" Oct 30 02:47:13.091117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.091093 5035 scope.go:122] "RemoveContainer" containerID="e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787" Oct 30 02:47:13.091450 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:13.091427 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787\": container with ID starting with e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787 not found: ID does not exist" containerID="e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787" Oct 30 02:47:13.091538 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.091463 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787"} err="failed to get container status \"e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787\": rpc error: code = NotFound desc = could not find container \"e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787\": container with ID starting with e3bb828bf1cd46e95621a50bb931560cd83f17135d70dafc3db383c5695a1787 not found: ID does not exist" Oct 30 02:47:13.122591 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.122551 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6"] Oct 30 02:47:13.126248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.126217 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-719029-66c86885dc-klbw6"] Oct 30 02:47:13.160166 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.160063 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz"] Oct 30 02:47:13.163991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.163947 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-719029-68f6795878-p8vdz"] Oct 30 02:47:13.167684 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.167660 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ht4f\" (UniqueName: \"kubernetes.io/projected/6f27a327-a17e-45e6-80fe-88d732edb488-kube-api-access-4ht4f\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:13.657594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.657469 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13653c11-c510-447b-90fb-2d959bee34d0" path="/var/lib/kubelet/pods/13653c11-c510-447b-90fb-2d959bee34d0/volumes" Oct 30 02:47:13.658797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.658756 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f27a327-a17e-45e6-80fe-88d732edb488" path="/var/lib/kubelet/pods/6f27a327-a17e-45e6-80fe-88d732edb488/volumes" Oct 30 02:47:13.808900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.808866 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz"] Oct 30 02:47:13.809117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:13.809091 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" containerID="cri-o://a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da" gracePeriod=30 Oct 30 02:47:14.821638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.821594 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp"] Oct 30 02:47:14.822287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822264 5035 cpu_manager.go:414] "containerMap: removing container" podUID="cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb" containerName="revoke-keys" Oct 30 02:47:14.822345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822290 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb" containerName="revoke-keys" Oct 30 02:47:14.822345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822304 5035 cpu_manager.go:414] "containerMap: removing container" podUID="6f27a327-a17e-45e6-80fe-88d732edb488" containerName="payload-pre-processing" Oct 30 02:47:14.822345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822310 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f27a327-a17e-45e6-80fe-88d732edb488" containerName="payload-pre-processing" Oct 30 02:47:14.822345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822325 5035 cpu_manager.go:414] "containerMap: removing container" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerName="cleanup" Oct 30 02:47:14.822345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822331 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8c42c5-d6c3-4f7c-afc2-14f672550351" containerName="cleanup" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822358 5035 cpu_manager.go:414] "containerMap: removing container" podUID="952801bc-170d-499d-83e1-781126e7cda2" containerName="cleanup" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822364 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952801bc-170d-499d-83e1-781126e7cda2" containerName="cleanup" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822370 5035 cpu_manager.go:414] "containerMap: removing container" podUID="13653c11-c510-447b-90fb-2d959bee34d0" containerName="payload-processing" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822375 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13653c11-c510-447b-90fb-2d959bee34d0" containerName="payload-processing" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822384 5035 cpu_manager.go:414] "containerMap: removing container" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerName="cleanup" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822388 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerName="cleanup" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822470 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="13653c11-c510-447b-90fb-2d959bee34d0" containerName="payload-processing" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822482 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8bd4e0e-ab0b-4804-966f-a79dbeb3d720" containerName="cleanup" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822490 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f7a1fb-dc50-4f51-ab08-e745a1a1703b" containerName="istio-proxy" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822497 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65263ba-75ac-4cd2-b89e-dbf5bd35988f" containerName="cleanup" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822506 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb" containerName="revoke-keys" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822511 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f27a327-a17e-45e6-80fe-88d732edb488" containerName="payload-pre-processing" Oct 30 02:47:14.822517 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.822522 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="952801bc-170d-499d-83e1-781126e7cda2" containerName="cleanup" Oct 30 02:47:14.827759 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.827738 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:14.845038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.844981 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:47:14.845494 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.845462 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:47:14.857064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.857028 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:47:14.857064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.857069 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:47:14.861140 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.861107 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp"] Oct 30 02:47:14.868146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.868098 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" podUID="a6bab1e7-a639-4904-a00f-05e71080b13e" containerName="payload-pre-processing" containerID="cri-o://feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664" gracePeriod=25 Oct 30 02:47:14.892213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.892163 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f09ebe-t79sp\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:14.892412 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.892222 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f09ebe-t79sp\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:14.892412 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.892341 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx89c\" (UniqueName: \"kubernetes.io/projected/0b168647-e1e7-45e3-8057-d38459e03d61-kube-api-access-qx89c\") pod \"maas-api-revoke-sub-87f09ebe-t79sp\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:14.993430 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.993378 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx89c\" (UniqueName: \"kubernetes.io/projected/0b168647-e1e7-45e3-8057-d38459e03d61-kube-api-access-qx89c\") pod \"maas-api-revoke-sub-87f09ebe-t79sp\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:14.993694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.993592 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f09ebe-t79sp\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:14.993694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.993633 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f09ebe-t79sp\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:14.994344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.994319 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f09ebe-t79sp\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:14.996114 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:14.996086 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f09ebe-t79sp\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:15.018397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.018365 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx89c\" (UniqueName: \"kubernetes.io/projected/0b168647-e1e7-45e3-8057-d38459e03d61-kube-api-access-qx89c\") pod \"maas-api-revoke-sub-87f09ebe-t79sp\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:15.141835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.141715 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:15.465473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.465383 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp"] Oct 30 02:47:15.473405 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:47:15.473293 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b168647_e1e7_45e3_8057_d38459e03d61.slice/crio-d21f984576cf6aa44cb1ea3193897c55baa496dfbe008f189327f4b4c1990adf WatchSource:0}: Error finding container d21f984576cf6aa44cb1ea3193897c55baa496dfbe008f189327f4b4c1990adf: Status 404 returned error can't find the container with id d21f984576cf6aa44cb1ea3193897c55baa496dfbe008f189327f4b4c1990adf Oct 30 02:47:15.497499 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.497463 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" containerID="cri-o://6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8" gracePeriod=45 Oct 30 02:47:15.504496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.504432 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" containerID="cri-o://c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af" gracePeriod=45 Oct 30 02:47:15.517506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.517459 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" containerID="cri-o://d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58" gracePeriod=45 Oct 30 02:47:15.701136 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.700067 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused" Oct 30 02:47:15.774441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.774407 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 02:47:15.808848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.804921 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-home\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-home\") pod \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " Oct 30 02:47:15.808848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.804986 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tmp-dir\") pod \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " Oct 30 02:47:15.808848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.805017 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-dshm\") pod \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " Oct 30 02:47:15.808848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.805068 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-model-cache\") pod \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " Oct 30 02:47:15.808848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.805106 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-kube-api-access-d9hbc\" (UniqueName: \"kubernetes.io/projected/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-kube-api-access-d9hbc\") pod \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " Oct 30 02:47:15.808848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.805199 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tls-certs\") pod \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\" (UID: \"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1\") " Oct 30 02:47:15.808848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.806221 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-home" pod "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" (UID: "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.808848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.806291 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-model-cache" pod "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" (UID: "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.808848 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.806611 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tmp-dir" pod "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" (UID: "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.809520 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.809346 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-dshm" pod "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" (UID: "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.809750 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.809717 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-kube-api-access-d9hbc" pod "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" (UID: "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1"). InnerVolumeSpecName "kube-api-access-d9hbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:15.811094 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.811067 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tls-certs" pod "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" (UID: "fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:15.841225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.841191 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 02:47:15.845543 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.845493 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 02:47:15.861777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.861733 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" podUID="350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:15.861958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.861797 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" podUID="02fd08d5-b5af-4b0c-9a55-e789cd75e036" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:15.907186 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907140 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-model-cache\") pod \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " Oct 30 02:47:15.907375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907240 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-dshm\") pod \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " Oct 30 02:47:15.907375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907281 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-tmp-dir\") pod \"388a3149-77ea-469e-8038-681c4f10cc37\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " Oct 30 02:47:15.907375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907343 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-home\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-home\") pod \"388a3149-77ea-469e-8038-681c4f10cc37\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " Oct 30 02:47:15.907542 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907405 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tls-certs\" (UniqueName: \"kubernetes.io/secret/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tls-certs\") pod \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " Oct 30 02:47:15.907542 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907461 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388a3149-77ea-469e-8038-681c4f10cc37-tls-certs\" (UniqueName: \"kubernetes.io/secret/388a3149-77ea-469e-8038-681c4f10cc37-tls-certs\") pod \"388a3149-77ea-469e-8038-681c4f10cc37\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " Oct 30 02:47:15.907542 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907468 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-model-cache" pod "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" (UID: "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.907718 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907551 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-kube-api-access-zxhvb\" (UniqueName: \"kubernetes.io/projected/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-kube-api-access-zxhvb\") pod \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " Oct 30 02:47:15.907718 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907599 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-tmp-dir" pod "388a3149-77ea-469e-8038-681c4f10cc37" (UID: "388a3149-77ea-469e-8038-681c4f10cc37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.907960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907721 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tmp-dir\") pod \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " Oct 30 02:47:15.907960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907744 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-home" pod "388a3149-77ea-469e-8038-681c4f10cc37" (UID: "388a3149-77ea-469e-8038-681c4f10cc37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.907960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907784 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-dshm\") pod \"388a3149-77ea-469e-8038-681c4f10cc37\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " Oct 30 02:47:15.907960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907858 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-model-cache\") pod \"388a3149-77ea-469e-8038-681c4f10cc37\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " Oct 30 02:47:15.907960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.907942 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-home\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-home\") pod \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\" (UID: \"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace\") " Oct 30 02:47:15.908345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908014 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388a3149-77ea-469e-8038-681c4f10cc37-kube-api-access-8bsbh\" (UniqueName: \"kubernetes.io/projected/388a3149-77ea-469e-8038-681c4f10cc37-kube-api-access-8bsbh\") pod \"388a3149-77ea-469e-8038-681c4f10cc37\" (UID: \"388a3149-77ea-469e-8038-681c4f10cc37\") " Oct 30 02:47:15.908345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908056 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tmp-dir" pod "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" (UID: "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.908345 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908282 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-model-cache" pod "388a3149-77ea-469e-8038-681c4f10cc37" (UID: "388a3149-77ea-469e-8038-681c4f10cc37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.908605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908581 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908611 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908625 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908637 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908653 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908665 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908678 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9hbc\" (UniqueName: \"kubernetes.io/projected/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-kube-api-access-d9hbc\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908692 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908707 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908719 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.908744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908730 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:15.909091 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.908833 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-home" pod "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" (UID: "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.910992 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.910955 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388a3149-77ea-469e-8038-681c4f10cc37-kube-api-access-8bsbh" pod "388a3149-77ea-469e-8038-681c4f10cc37" (UID: "388a3149-77ea-469e-8038-681c4f10cc37"). InnerVolumeSpecName "kube-api-access-8bsbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:15.911244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.911223 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-dshm" pod "388a3149-77ea-469e-8038-681c4f10cc37" (UID: "388a3149-77ea-469e-8038-681c4f10cc37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.911829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.911800 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-dshm" pod "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" (UID: "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:15.911829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.911812 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388a3149-77ea-469e-8038-681c4f10cc37-tls-certs" pod "388a3149-77ea-469e-8038-681c4f10cc37" (UID: "388a3149-77ea-469e-8038-681c4f10cc37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:15.912022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.911807 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-kube-api-access-zxhvb" pod "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" (UID: "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace"). InnerVolumeSpecName "kube-api-access-zxhvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:15.913316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:15.913294 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tls-certs" pod "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" (UID: "90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:16.011469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.011279 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:16.011469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.011370 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:16.011469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.011395 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388a3149-77ea-469e-8038-681c4f10cc37-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:16.011469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.011420 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxhvb\" (UniqueName: \"kubernetes.io/projected/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-kube-api-access-zxhvb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:16.011469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.011447 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388a3149-77ea-469e-8038-681c4f10cc37-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:16.011469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.011466 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:16.012091 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.011488 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bsbh\" (UniqueName: \"kubernetes.io/projected/388a3149-77ea-469e-8038-681c4f10cc37-kube-api-access-8bsbh\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:16.108826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.108729 5035 generic.go:356] "Generic (PLEG): container finished" podID="388a3149-77ea-469e-8038-681c4f10cc37" containerID="d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58" exitCode=0 Oct 30 02:47:16.109189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.108830 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" event={"ID":"388a3149-77ea-469e-8038-681c4f10cc37","Type":"ContainerDied","Data":"d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58"} Oct 30 02:47:16.109189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.108889 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" Oct 30 02:47:16.109189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.108941 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z" event={"ID":"388a3149-77ea-469e-8038-681c4f10cc37","Type":"ContainerDied","Data":"dd017b63b127254cecf5719574e4ebaea0f8eb56e8719c54b752d0ea0299595c"} Oct 30 02:47:16.109189 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.108975 5035 scope.go:122] "RemoveContainer" containerID="d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58" Oct 30 02:47:16.111077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.111043 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" event={"ID":"0b168647-e1e7-45e3-8057-d38459e03d61","Type":"ContainerStarted","Data":"097a77af8f4183a9eabadc34f713b01fec3af0a586301ac92c5b5284a94b2ab7"} Oct 30 02:47:16.111226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.111139 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" event={"ID":"0b168647-e1e7-45e3-8057-d38459e03d61","Type":"ContainerStarted","Data":"d21f984576cf6aa44cb1ea3193897c55baa496dfbe008f189327f4b4c1990adf"} Oct 30 02:47:16.112843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.112817 5035 generic.go:356] "Generic (PLEG): container finished" podID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerID="c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af" exitCode=0 Oct 30 02:47:16.112979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.112863 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" Oct 30 02:47:16.112979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.112896 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" event={"ID":"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace","Type":"ContainerDied","Data":"c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af"} Oct 30 02:47:16.112979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.112940 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946" event={"ID":"90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace","Type":"ContainerDied","Data":"b648122c6d8033fdd3189f2615146e5bfd88794d23128faafca043b0f9491dac"} Oct 30 02:47:16.114542 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.114504 5035 generic.go:356] "Generic (PLEG): container finished" podID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerID="6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8" exitCode=0 Oct 30 02:47:16.114709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.114688 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" event={"ID":"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1","Type":"ContainerDied","Data":"6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8"} Oct 30 02:47:16.114774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.114723 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" event={"ID":"fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1","Type":"ContainerDied","Data":"80abb30f24d3bcae2ac87611226334c5d7af723514175a56d5efcdb0fd8dc65d"} Oct 30 02:47:16.114823 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.114784 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn" Oct 30 02:47:16.122197 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.122174 5035 scope.go:122] "RemoveContainer" containerID="d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58" Oct 30 02:47:16.122572 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:16.122513 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58\": container with ID starting with d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58 not found: ID does not exist" containerID="d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58" Oct 30 02:47:16.122669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.122582 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58"} err="failed to get container status \"d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58\": rpc error: code = NotFound desc = could not find container \"d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58\": container with ID starting with d735063e2ba88c9ed057f69187556853081109d0bbdbd5e603ff6e7bb8c42b58 not found: ID does not exist" Oct 30 02:47:16.122669 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.122600 5035 scope.go:122] "RemoveContainer" containerID="c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af" Oct 30 02:47:16.132605 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.132580 5035 scope.go:122] "RemoveContainer" containerID="c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af" Oct 30 02:47:16.132938 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:16.132914 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af\": container with ID starting with c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af not found: ID does not exist" containerID="c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af" Oct 30 02:47:16.133020 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.132947 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af"} err="failed to get container status \"c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af\": rpc error: code = NotFound desc = could not find container \"c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af\": container with ID starting with c500919d863da3f2fe200c850fadd4eb6573b8467b28f824806fd9b2cca2c7af not found: ID does not exist" Oct 30 02:47:16.133020 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.132967 5035 scope.go:122] "RemoveContainer" containerID="6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8" Oct 30 02:47:16.142475 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.142453 5035 scope.go:122] "RemoveContainer" containerID="6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8" Oct 30 02:47:16.142792 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:16.142766 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8\": container with ID starting with 6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8 not found: ID does not exist" containerID="6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8" Oct 30 02:47:16.142855 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.142801 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8"} err="failed to get container status \"6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8\": rpc error: code = NotFound desc = could not find container \"6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8\": container with ID starting with 6ff6e44916b1e77f5f13e2b3afda5dea9df231267860bcf92d60349e3f73adb8 not found: ID does not exist" Oct 30 02:47:16.182564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.182474 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" podStartSLOduration=2.18245866 podStartE2EDuration="2.18245866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:47:16.180624188 +0000 UTC m=+4714.683556409" watchObservedRunningTime="2026-10-30 02:47:16.18245866 +0000 UTC m=+4714.685390879" Oct 30 02:47:16.235821 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.235775 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn"] Oct 30 02:47:16.238434 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.238398 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-719029/distinct-w2-719029-kserve-7b54b578c5-mvkpn"] Oct 30 02:47:16.273825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.273737 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946"] Oct 30 02:47:16.279356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.279323 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-719029/premium-w2-719029-kserve-6c6b979dc6-6h946"] Oct 30 02:47:16.314550 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.314303 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z"] Oct 30 02:47:16.319444 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:16.319406 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-719029/simulator-w2-719029-kserve-6b5544794f-vnf6z"] Oct 30 02:47:17.126070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:17.126035 5035 generic.go:356] "Generic (PLEG): container finished" podID="0b168647-e1e7-45e3-8057-d38459e03d61" containerID="097a77af8f4183a9eabadc34f713b01fec3af0a586301ac92c5b5284a94b2ab7" exitCode=0 Oct 30 02:47:17.126632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:17.126123 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" event={"ID":"0b168647-e1e7-45e3-8057-d38459e03d61","Type":"ContainerDied","Data":"097a77af8f4183a9eabadc34f713b01fec3af0a586301ac92c5b5284a94b2ab7"} Oct 30 02:47:17.629206 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:17.629160 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:47:17.647065 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:17.647010 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388a3149-77ea-469e-8038-681c4f10cc37" path="/var/lib/kubelet/pods/388a3149-77ea-469e-8038-681c4f10cc37/volumes" Oct 30 02:47:17.647849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:17.647830 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" path="/var/lib/kubelet/pods/90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace/volumes" Oct 30 02:47:17.648357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:17.648341 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" path="/var/lib/kubelet/pods/fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1/volumes" Oct 30 02:47:18.295545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.295493 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:18.339577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.339423 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-xks-ca\") pod \"0b168647-e1e7-45e3-8057-d38459e03d61\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " Oct 30 02:47:18.340333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.339750 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b168647-e1e7-45e3-8057-d38459e03d61-kube-api-access-qx89c\" (UniqueName: \"kubernetes.io/projected/0b168647-e1e7-45e3-8057-d38459e03d61-kube-api-access-qx89c\") pod \"0b168647-e1e7-45e3-8057-d38459e03d61\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " Oct 30 02:47:18.340333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.339957 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-openshift-ca\") pod \"0b168647-e1e7-45e3-8057-d38459e03d61\" (UID: \"0b168647-e1e7-45e3-8057-d38459e03d61\") " Oct 30 02:47:18.342169 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.341968 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-openshift-ca" pod "0b168647-e1e7-45e3-8057-d38459e03d61" (UID: "0b168647-e1e7-45e3-8057-d38459e03d61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:18.347715 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.347578 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b168647-e1e7-45e3-8057-d38459e03d61-kube-api-access-qx89c" pod "0b168647-e1e7-45e3-8057-d38459e03d61" (UID: "0b168647-e1e7-45e3-8057-d38459e03d61"). InnerVolumeSpecName "kube-api-access-qx89c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:18.348270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.348141 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-xks-ca" pod "0b168647-e1e7-45e3-8057-d38459e03d61" (UID: "0b168647-e1e7-45e3-8057-d38459e03d61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:18.442124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.442025 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:18.442124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.442064 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx89c\" (UniqueName: \"kubernetes.io/projected/0b168647-e1e7-45e3-8057-d38459e03d61-kube-api-access-qx89c\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:18.442124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:18.442074 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b168647-e1e7-45e3-8057-d38459e03d61-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.077685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.077615 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:47:19.147020 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.146981 5035 generic.go:356] "Generic (PLEG): container finished" podID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerID="a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da" exitCode=0 Oct 30 02:47:19.147190 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.147153 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" Oct 30 02:47:19.147262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.147200 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" event={"ID":"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3","Type":"ContainerDied","Data":"a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da"} Oct 30 02:47:19.147394 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.147262 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz" event={"ID":"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3","Type":"ContainerDied","Data":"32c7b5362736ef7ade892d01f85cc4d713e9f32a79fcb324452fb9076ccafc5c"} Oct 30 02:47:19.147394 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.147292 5035 scope.go:122] "RemoveContainer" containerID="a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da" Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.149403 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-certs\") pod \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.149454 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-podinfo\") pod \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.149964 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-socket" pod "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" (UID: "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.150351 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-certs" pod "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" (UID: "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.150460 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" event={"ID":"0b168647-e1e7-45e3-8057-d38459e03d61","Type":"ContainerDied","Data":"d21f984576cf6aa44cb1ea3193897c55baa496dfbe008f189327f4b4c1990adf"} Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.150501 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp" Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.150510 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d21f984576cf6aa44cb1ea3193897c55baa496dfbe008f189327f4b4c1990adf" Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151127 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-socket\") pod \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151230 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-credential-socket\") pod \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151271 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-token\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-token\") pod \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151311 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-envoy\") pod \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151370 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-data\") pod \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151422 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istiod-ca-cert\") pod \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151499 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-kube-api-access-ttkpb\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-kube-api-access-ttkpb\") pod \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\" (UID: \"f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3\") " Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151507 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-credential-socket" pod "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" (UID: "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151940 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151964 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.151978 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.152834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.152789 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-data" pod "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" (UID: "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:19.154309 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.153030 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-podinfo" pod "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" (UID: "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:47:19.154309 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.153091 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istiod-ca-cert" pod "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" (UID: "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:19.154956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.154927 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-kube-api-access-ttkpb" pod "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" (UID: "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3"). InnerVolumeSpecName "kube-api-access-ttkpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:19.157681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.157658 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-token" pod "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" (UID: "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:19.163058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.163034 5035 scope.go:122] "RemoveContainer" containerID="a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da" Oct 30 02:47:19.163451 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:19.163429 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da\": container with ID starting with a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da not found: ID does not exist" containerID="a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da" Oct 30 02:47:19.163522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.163463 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da"} err="failed to get container status \"a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da\": rpc error: code = NotFound desc = could not find container \"a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da\": container with ID starting with a61ac38e67dcd57dbb2d64df3e3c2eebed24987e154c5ab136dedf4536dc40da not found: ID does not exist" Oct 30 02:47:19.164069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.164040 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-envoy" pod "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" (UID: "f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:19.252638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.252595 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.252638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.252630 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.252638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.252639 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttkpb\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-kube-api-access-ttkpb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.252638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.252649 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.252940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.252659 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.252940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.252667 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:19.434175 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.434085 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" podUID="a6bab1e7-a639-4904-a00f-05e71080b13e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:19.494421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.494373 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz"] Oct 30 02:47:19.500963 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.500924 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-594253-openshift-default-fc44dff6c-zftfz"] Oct 30 02:47:19.642708 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:19.642674 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" path="/var/lib/kubelet/pods/f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3/volumes" Oct 30 02:47:22.637482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.637436 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8"] Oct 30 02:47:22.638040 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638021 5035 cpu_manager.go:414] "containerMap: removing container" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" Oct 30 02:47:22.638040 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638040 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" Oct 30 02:47:22.638121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638054 5035 cpu_manager.go:414] "containerMap: removing container" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" Oct 30 02:47:22.638121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638059 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" Oct 30 02:47:22.638121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638077 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0b168647-e1e7-45e3-8057-d38459e03d61" containerName="revoke-keys" Oct 30 02:47:22.638121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638083 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b168647-e1e7-45e3-8057-d38459e03d61" containerName="revoke-keys" Oct 30 02:47:22.638121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638101 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" Oct 30 02:47:22.638121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638105 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" Oct 30 02:47:22.638121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638116 5035 cpu_manager.go:414] "containerMap: removing container" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" Oct 30 02:47:22.638121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638121 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" Oct 30 02:47:22.638372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638197 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b168647-e1e7-45e3-8057-d38459e03d61" containerName="revoke-keys" Oct 30 02:47:22.638372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638208 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a935d8-06c6-43cb-bbc6-adf7b9f60dc3" containerName="istio-proxy" Oct 30 02:47:22.638372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638215 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbca3d1-d147-4558-aa31-9f7f6fd0fcd1" containerName="main" Oct 30 02:47:22.638372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638223 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d83b6a-03b9-4fd9-8fa9-74cf8d8e7ace" containerName="main" Oct 30 02:47:22.638372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.638228 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="388a3149-77ea-469e-8038-681c4f10cc37" containerName="main" Oct 30 02:47:22.642931 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.642913 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.672021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.671988 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:47:22.672021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.671993 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:47:22.672021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.672002 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:47:22.672363 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.672100 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:47:22.674874 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.674839 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8"] Oct 30 02:47:22.797486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.797432 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5cpt\" (UniqueName: \"kubernetes.io/projected/1f439cc0-719a-4a9d-9881-24605b04af37-kube-api-access-r5cpt\") pod \"maas-api-revoke-sub-69cefe28-j9lk8\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.797753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.797499 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69cefe28-j9lk8\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.797753 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.797619 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69cefe28-j9lk8\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.898735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.898593 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69cefe28-j9lk8\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.898928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.898736 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5cpt\" (UniqueName: \"kubernetes.io/projected/1f439cc0-719a-4a9d-9881-24605b04af37-kube-api-access-r5cpt\") pod \"maas-api-revoke-sub-69cefe28-j9lk8\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.898928 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.898772 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69cefe28-j9lk8\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.899377 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.899347 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69cefe28-j9lk8\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.901098 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.901078 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69cefe28-j9lk8\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.922212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.922178 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5cpt\" (UniqueName: \"kubernetes.io/projected/1f439cc0-719a-4a9d-9881-24605b04af37-kube-api-access-r5cpt\") pod \"maas-api-revoke-sub-69cefe28-j9lk8\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:22.957423 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:22.957387 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:23.265500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:23.265472 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8"] Oct 30 02:47:23.267782 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:47:23.267742 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f439cc0_719a_4a9d_9881_24605b04af37.slice/crio-9461dfa75cb84deb06b4f7758e1f128e7925acf9f0485a87ff2c42376fe20b0a WatchSource:0}: Error finding container 9461dfa75cb84deb06b4f7758e1f128e7925acf9f0485a87ff2c42376fe20b0a: Status 404 returned error can't find the container with id 9461dfa75cb84deb06b4f7758e1f128e7925acf9f0485a87ff2c42376fe20b0a Oct 30 02:47:24.184831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:24.184799 5035 generic.go:356] "Generic (PLEG): container finished" podID="1f439cc0-719a-4a9d-9881-24605b04af37" containerID="6df51f5bc9da0d440c4f45ac0c0bdc75dc3894881906600acbfae1e8c2cf4b7d" exitCode=0 Oct 30 02:47:24.185301 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:24.184889 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" event={"ID":"1f439cc0-719a-4a9d-9881-24605b04af37","Type":"ContainerDied","Data":"6df51f5bc9da0d440c4f45ac0c0bdc75dc3894881906600acbfae1e8c2cf4b7d"} Oct 30 02:47:24.185301 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:24.184957 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" event={"ID":"1f439cc0-719a-4a9d-9881-24605b04af37","Type":"ContainerStarted","Data":"9461dfa75cb84deb06b4f7758e1f128e7925acf9f0485a87ff2c42376fe20b0a"} Oct 30 02:47:25.373786 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.373758 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:25.530624 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.530582 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f439cc0-719a-4a9d-9881-24605b04af37-kube-api-access-r5cpt\" (UniqueName: \"kubernetes.io/projected/1f439cc0-719a-4a9d-9881-24605b04af37-kube-api-access-r5cpt\") pod \"1f439cc0-719a-4a9d-9881-24605b04af37\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " Oct 30 02:47:25.530849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.530684 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-xks-ca\") pod \"1f439cc0-719a-4a9d-9881-24605b04af37\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " Oct 30 02:47:25.530849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.530705 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-openshift-ca\") pod \"1f439cc0-719a-4a9d-9881-24605b04af37\" (UID: \"1f439cc0-719a-4a9d-9881-24605b04af37\") " Oct 30 02:47:25.531174 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.531140 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-openshift-ca" pod "1f439cc0-719a-4a9d-9881-24605b04af37" (UID: "1f439cc0-719a-4a9d-9881-24605b04af37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:25.532968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.532936 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-xks-ca" pod "1f439cc0-719a-4a9d-9881-24605b04af37" (UID: "1f439cc0-719a-4a9d-9881-24605b04af37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:25.533059 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.532992 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f439cc0-719a-4a9d-9881-24605b04af37-kube-api-access-r5cpt" pod "1f439cc0-719a-4a9d-9881-24605b04af37" (UID: "1f439cc0-719a-4a9d-9881-24605b04af37"). InnerVolumeSpecName "kube-api-access-r5cpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:25.632385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.632347 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5cpt\" (UniqueName: \"kubernetes.io/projected/1f439cc0-719a-4a9d-9881-24605b04af37-kube-api-access-r5cpt\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:25.632385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.632383 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:25.632385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.632394 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f439cc0-719a-4a9d-9881-24605b04af37-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:25.862014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.861922 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" podUID="02fd08d5-b5af-4b0c-9a55-e789cd75e036" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:25.862163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:25.862011 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" podUID="350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:26.198864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.198775 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" Oct 30 02:47:26.198864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.198808 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8" event={"ID":"1f439cc0-719a-4a9d-9881-24605b04af37","Type":"ContainerDied","Data":"9461dfa75cb84deb06b4f7758e1f128e7925acf9f0485a87ff2c42376fe20b0a"} Oct 30 02:47:26.198864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.198859 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9461dfa75cb84deb06b4f7758e1f128e7925acf9f0485a87ff2c42376fe20b0a" Oct 30 02:47:26.361930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.361891 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr"] Oct 30 02:47:26.362436 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.362418 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1f439cc0-719a-4a9d-9881-24605b04af37" containerName="revoke-keys" Oct 30 02:47:26.362492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.362438 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f439cc0-719a-4a9d-9881-24605b04af37" containerName="revoke-keys" Oct 30 02:47:26.362572 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.362562 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f439cc0-719a-4a9d-9881-24605b04af37" containerName="revoke-keys" Oct 30 02:47:26.365807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.365785 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.383497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.383460 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:47:26.384414 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.383779 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:47:26.384414 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.384003 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:47:26.388026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.388001 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:47:26.391476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.391448 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr"] Oct 30 02:47:26.545119 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.545073 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d9527a-xp2qr\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.545399 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.545149 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d9527a-xp2qr\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.545399 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.545247 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4qx\" (UniqueName: \"kubernetes.io/projected/027b1899-8034-49b9-9ae9-6a29f15a03f8-kube-api-access-pl4qx\") pod \"maas-api-revoke-sub-88d9527a-xp2qr\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.641480 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.641442 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6"] Oct 30 02:47:26.646121 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.646092 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d9527a-xp2qr\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.646232 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.646159 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d9527a-xp2qr\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.646232 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.646190 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4qx\" (UniqueName: \"kubernetes.io/projected/027b1899-8034-49b9-9ae9-6a29f15a03f8-kube-api-access-pl4qx\") pod \"maas-api-revoke-sub-88d9527a-xp2qr\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.647140 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.647114 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d9527a-xp2qr\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.648965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.648929 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d9527a-xp2qr\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.669362 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.669318 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4qx\" (UniqueName: \"kubernetes.io/projected/027b1899-8034-49b9-9ae9-6a29f15a03f8-kube-api-access-pl4qx\") pod \"maas-api-revoke-sub-88d9527a-xp2qr\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.674427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.674381 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6"] Oct 30 02:47:26.681331 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.681300 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:26.995974 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:26.995944 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr"] Oct 30 02:47:26.997828 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:47:26.997798 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027b1899_8034_49b9_9ae9_6a29f15a03f8.slice/crio-0ed42c7a60fb2d4f71693cb2144a8f83c5d02414a8c7e4f8f51430ca8bf5f27b WatchSource:0}: Error finding container 0ed42c7a60fb2d4f71693cb2144a8f83c5d02414a8c7e4f8f51430ca8bf5f27b: Status 404 returned error can't find the container with id 0ed42c7a60fb2d4f71693cb2144a8f83c5d02414a8c7e4f8f51430ca8bf5f27b Oct 30 02:47:27.165401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:27.165360 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns"] Oct 30 02:47:27.206143 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:27.206107 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" event={"ID":"027b1899-8034-49b9-9ae9-6a29f15a03f8","Type":"ContainerStarted","Data":"0ed42c7a60fb2d4f71693cb2144a8f83c5d02414a8c7e4f8f51430ca8bf5f27b"} Oct 30 02:47:28.214589 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.214523 5035 generic.go:356] "Generic (PLEG): container finished" podID="027b1899-8034-49b9-9ae9-6a29f15a03f8" containerID="fd747100c7c28d7edea9be3077fb4526a223fb2b77e58ac645a51f0bcc2027f8" exitCode=0 Oct 30 02:47:28.215001 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.214558 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" event={"ID":"027b1899-8034-49b9-9ae9-6a29f15a03f8","Type":"ContainerDied","Data":"fd747100c7c28d7edea9be3077fb4526a223fb2b77e58ac645a51f0bcc2027f8"} Oct 30 02:47:28.778485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.778458 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:47:28.814889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.814855 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:47:28.869596 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.869552 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02fd08d5-b5af-4b0c-9a55-e789cd75e036-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fd08d5-b5af-4b0c-9a55-e789cd75e036-plugins-config-volume\") pod \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\" (UID: \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\") " Oct 30 02:47:28.869781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.869617 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02fd08d5-b5af-4b0c-9a55-e789cd75e036-kube-api-access-b6xjk\" (UniqueName: \"kubernetes.io/projected/02fd08d5-b5af-4b0c-9a55-e789cd75e036-kube-api-access-b6xjk\") pod \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\" (UID: \"02fd08d5-b5af-4b0c-9a55-e789cd75e036\") " Oct 30 02:47:28.869909 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.869884 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fd08d5-b5af-4b0c-9a55-e789cd75e036-plugins-config-volume" pod "02fd08d5-b5af-4b0c-9a55-e789cd75e036" (UID: "02fd08d5-b5af-4b0c-9a55-e789cd75e036"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:28.870196 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.870169 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fd08d5-b5af-4b0c-9a55-e789cd75e036-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:28.872054 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.872023 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fd08d5-b5af-4b0c-9a55-e789cd75e036-kube-api-access-b6xjk" pod "02fd08d5-b5af-4b0c-9a55-e789cd75e036" (UID: "02fd08d5-b5af-4b0c-9a55-e789cd75e036"). InnerVolumeSpecName "kube-api-access-b6xjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:28.971522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.971483 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-kube-api-access-8t4r4\" (UniqueName: \"kubernetes.io/projected/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-kube-api-access-8t4r4\") pod \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\" (UID: \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\") " Oct 30 02:47:28.971738 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.971613 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-plugins-config-volume\") pod \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\" (UID: \"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad\") " Oct 30 02:47:28.972048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.972020 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-plugins-config-volume" pod "350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" (UID: "350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:28.972126 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.972088 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6xjk\" (UniqueName: \"kubernetes.io/projected/02fd08d5-b5af-4b0c-9a55-e789cd75e036-kube-api-access-b6xjk\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:28.973884 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:28.973856 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-kube-api-access-8t4r4" pod "350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" (UID: "350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad"). InnerVolumeSpecName "kube-api-access-8t4r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:29.073185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.073090 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t4r4\" (UniqueName: \"kubernetes.io/projected/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-kube-api-access-8t4r4\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:29.073185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.073124 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:29.222311 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.222272 5035 generic.go:356] "Generic (PLEG): container finished" podID="350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" containerID="3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541" exitCode=0 Oct 30 02:47:29.222860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.222335 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" Oct 30 02:47:29.222860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.222357 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" event={"ID":"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad","Type":"ContainerDied","Data":"3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541"} Oct 30 02:47:29.222860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.222417 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj" event={"ID":"350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad","Type":"ContainerDied","Data":"898d7e39f725f88afaf125e2dfc5bc6012169486520923e0f1aa9b6a6677a1b8"} Oct 30 02:47:29.222860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.222442 5035 scope.go:122] "RemoveContainer" containerID="3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541" Oct 30 02:47:29.223825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.223801 5035 generic.go:356] "Generic (PLEG): container finished" podID="02fd08d5-b5af-4b0c-9a55-e789cd75e036" containerID="d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9" exitCode=0 Oct 30 02:47:29.223929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.223855 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" Oct 30 02:47:29.223929 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.223894 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" event={"ID":"02fd08d5-b5af-4b0c-9a55-e789cd75e036","Type":"ContainerDied","Data":"d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9"} Oct 30 02:47:29.224049 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.223959 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9" event={"ID":"02fd08d5-b5af-4b0c-9a55-e789cd75e036","Type":"ContainerDied","Data":"d3b88cc98aa45ac0cede693e72ca5a403489ccde57d8c13cb74f6e2a10bfb53c"} Oct 30 02:47:29.234545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.234507 5035 scope.go:122] "RemoveContainer" containerID="3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541" Oct 30 02:47:29.234954 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:29.234922 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541\": container with ID starting with 3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541 not found: ID does not exist" containerID="3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541" Oct 30 02:47:29.235073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.234966 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541"} err="failed to get container status \"3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541\": rpc error: code = NotFound desc = could not find container \"3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541\": container with ID starting with 3f6524ba7d33dd0f682cb3d0de66ae62ca49f7bf5ceaf9883109c9d954d1e541 not found: ID does not exist" Oct 30 02:47:29.235073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.234989 5035 scope.go:122] "RemoveContainer" containerID="d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9" Oct 30 02:47:29.247570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.247520 5035 scope.go:122] "RemoveContainer" containerID="d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9" Oct 30 02:47:29.247934 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:29.247898 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9\": container with ID starting with d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9 not found: ID does not exist" containerID="d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9" Oct 30 02:47:29.248001 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.247950 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9"} err="failed to get container status \"d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9\": rpc error: code = NotFound desc = could not find container \"d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9\": container with ID starting with d19af64a7298bbf433a3d78744eea7b5fd7ce4f9639becd161a979b186222ed9 not found: ID does not exist" Oct 30 02:47:29.273252 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.273213 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj"] Oct 30 02:47:29.286139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.286101 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-594253-9c78f8587-4hdwj"] Oct 30 02:47:29.320606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.320566 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9"] Oct 30 02:47:29.329131 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.329037 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-594253-67c8c9db64-58zp9"] Oct 30 02:47:29.365837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.365808 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:29.434087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.434041 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" podUID="a6bab1e7-a639-4904-a00f-05e71080b13e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:29.476569 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.476520 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-openshift-ca\") pod \"027b1899-8034-49b9-9ae9-6a29f15a03f8\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " Oct 30 02:47:29.476767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.476643 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-xks-ca\") pod \"027b1899-8034-49b9-9ae9-6a29f15a03f8\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " Oct 30 02:47:29.476767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.476744 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027b1899-8034-49b9-9ae9-6a29f15a03f8-kube-api-access-pl4qx\" (UniqueName: \"kubernetes.io/projected/027b1899-8034-49b9-9ae9-6a29f15a03f8-kube-api-access-pl4qx\") pod \"027b1899-8034-49b9-9ae9-6a29f15a03f8\" (UID: \"027b1899-8034-49b9-9ae9-6a29f15a03f8\") " Oct 30 02:47:29.476995 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.476966 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-openshift-ca" pod "027b1899-8034-49b9-9ae9-6a29f15a03f8" (UID: "027b1899-8034-49b9-9ae9-6a29f15a03f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:29.477130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.477114 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:29.478921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.478892 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-xks-ca" pod "027b1899-8034-49b9-9ae9-6a29f15a03f8" (UID: "027b1899-8034-49b9-9ae9-6a29f15a03f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:29.479021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.478969 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027b1899-8034-49b9-9ae9-6a29f15a03f8-kube-api-access-pl4qx" pod "027b1899-8034-49b9-9ae9-6a29f15a03f8" (UID: "027b1899-8034-49b9-9ae9-6a29f15a03f8"). InnerVolumeSpecName "kube-api-access-pl4qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:29.577819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.577778 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl4qx\" (UniqueName: \"kubernetes.io/projected/027b1899-8034-49b9-9ae9-6a29f15a03f8-kube-api-access-pl4qx\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:29.577819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.577812 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027b1899-8034-49b9-9ae9-6a29f15a03f8-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:29.641691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.641607 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02fd08d5-b5af-4b0c-9a55-e789cd75e036" path="/var/lib/kubelet/pods/02fd08d5-b5af-4b0c-9a55-e789cd75e036/volumes" Oct 30 02:47:29.642020 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:29.642007 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" path="/var/lib/kubelet/pods/350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad/volumes" Oct 30 02:47:30.233827 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:30.233782 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" event={"ID":"027b1899-8034-49b9-9ae9-6a29f15a03f8","Type":"ContainerDied","Data":"0ed42c7a60fb2d4f71693cb2144a8f83c5d02414a8c7e4f8f51430ca8bf5f27b"} Oct 30 02:47:30.234269 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:30.233842 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ed42c7a60fb2d4f71693cb2144a8f83c5d02414a8c7e4f8f51430ca8bf5f27b" Oct 30 02:47:30.234269 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:30.233808 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr" Oct 30 02:47:31.642207 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:31.642159 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" podUID="1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" containerName="payload-processing" containerID="cri-o://ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d" gracePeriod=25 Oct 30 02:47:31.675299 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:31.675253 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" podUID="00e5e32e-4e1a-4585-967c-d98ad0617ec9" containerName="payload-pre-processing" containerID="cri-o://dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548" gracePeriod=25 Oct 30 02:47:32.775730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:32.775679 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" podUID="1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:32.776161 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:32.775913 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" podUID="00e5e32e-4e1a-4585-967c-d98ad0617ec9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:33.881271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:33.881224 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6"] Oct 30 02:47:33.911188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:33.911143 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl"] Oct 30 02:47:34.289104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.289049 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt"] Oct 30 02:47:34.289914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.289882 5035 cpu_manager.go:414] "containerMap: removing container" podUID="027b1899-8034-49b9-9ae9-6a29f15a03f8" containerName="revoke-keys" Oct 30 02:47:34.289914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.289915 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027b1899-8034-49b9-9ae9-6a29f15a03f8" containerName="revoke-keys" Oct 30 02:47:34.290128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.289956 5035 cpu_manager.go:414] "containerMap: removing container" podUID="350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" containerName="payload-pre-processing" Oct 30 02:47:34.290128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.289962 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" containerName="payload-pre-processing" Oct 30 02:47:34.290128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.289980 5035 cpu_manager.go:414] "containerMap: removing container" podUID="02fd08d5-b5af-4b0c-9a55-e789cd75e036" containerName="payload-processing" Oct 30 02:47:34.290128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.289986 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02fd08d5-b5af-4b0c-9a55-e789cd75e036" containerName="payload-processing" Oct 30 02:47:34.290128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.290070 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="350c87ad-eb3c-4b12-bdf4-15bf9ef6a7ad" containerName="payload-pre-processing" Oct 30 02:47:34.290128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.290083 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="02fd08d5-b5af-4b0c-9a55-e789cd75e036" containerName="payload-processing" Oct 30 02:47:34.290128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.290092 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="027b1899-8034-49b9-9ae9-6a29f15a03f8" containerName="revoke-keys" Oct 30 02:47:34.294873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.294846 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.311603 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.311565 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:47:34.311804 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.311682 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:47:34.311804 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.311764 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:47:34.311951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.311807 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:47:34.314943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.314903 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt"] Oct 30 02:47:34.328039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.327994 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849af60a-jj8zt\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.328228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.328124 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbpq\" (UniqueName: \"kubernetes.io/projected/3d626614-4ffd-4020-a2c6-987a96bd21dd-kube-api-access-qpbpq\") pod \"maas-api-revoke-sub-849af60a-jj8zt\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.328303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.328284 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849af60a-jj8zt\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.430091 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.430033 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849af60a-jj8zt\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.430341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.430170 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849af60a-jj8zt\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.430341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.430243 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbpq\" (UniqueName: \"kubernetes.io/projected/3d626614-4ffd-4020-a2c6-987a96bd21dd-kube-api-access-qpbpq\") pod \"maas-api-revoke-sub-849af60a-jj8zt\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.432062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.432031 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849af60a-jj8zt\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.434740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.434713 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849af60a-jj8zt\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.453641 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.453605 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbpq\" (UniqueName: \"kubernetes.io/projected/3d626614-4ffd-4020-a2c6-987a96bd21dd-kube-api-access-qpbpq\") pod \"maas-api-revoke-sub-849af60a-jj8zt\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.610620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.610490 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:34.927446 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:34.927418 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt"] Oct 30 02:47:34.966801 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:47:34.966764 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d626614_4ffd_4020_a2c6_987a96bd21dd.slice/crio-3ab62b9fe983ce0daf390c441ef843116b55a2e2aa26241b0b170a9d23d23203 WatchSource:0}: Error finding container 3ab62b9fe983ce0daf390c441ef843116b55a2e2aa26241b0b170a9d23d23203: Status 404 returned error can't find the container with id 3ab62b9fe983ce0daf390c441ef843116b55a2e2aa26241b0b170a9d23d23203 Oct 30 02:47:35.121224 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.121195 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:47:35.138180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.138062 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6bab1e7-a639-4904-a00f-05e71080b13e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bab1e7-a639-4904-a00f-05e71080b13e-plugins-config-volume\") pod \"a6bab1e7-a639-4904-a00f-05e71080b13e\" (UID: \"a6bab1e7-a639-4904-a00f-05e71080b13e\") " Oct 30 02:47:35.138404 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.138241 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6bab1e7-a639-4904-a00f-05e71080b13e-kube-api-access-92kbt\" (UniqueName: \"kubernetes.io/projected/a6bab1e7-a639-4904-a00f-05e71080b13e-kube-api-access-92kbt\") pod \"a6bab1e7-a639-4904-a00f-05e71080b13e\" (UID: \"a6bab1e7-a639-4904-a00f-05e71080b13e\") " Oct 30 02:47:35.138571 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.138546 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6bab1e7-a639-4904-a00f-05e71080b13e-plugins-config-volume" pod "a6bab1e7-a639-4904-a00f-05e71080b13e" (UID: "a6bab1e7-a639-4904-a00f-05e71080b13e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:35.138709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.138677 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bab1e7-a639-4904-a00f-05e71080b13e-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:35.140693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.140661 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bab1e7-a639-4904-a00f-05e71080b13e-kube-api-access-92kbt" pod "a6bab1e7-a639-4904-a00f-05e71080b13e" (UID: "a6bab1e7-a639-4904-a00f-05e71080b13e"). InnerVolumeSpecName "kube-api-access-92kbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:35.239933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.239882 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92kbt\" (UniqueName: \"kubernetes.io/projected/a6bab1e7-a639-4904-a00f-05e71080b13e-kube-api-access-92kbt\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:35.265422 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.265385 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" event={"ID":"3d626614-4ffd-4020-a2c6-987a96bd21dd","Type":"ContainerStarted","Data":"3ab62b9fe983ce0daf390c441ef843116b55a2e2aa26241b0b170a9d23d23203"} Oct 30 02:47:35.266831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.266794 5035 generic.go:356] "Generic (PLEG): container finished" podID="a6bab1e7-a639-4904-a00f-05e71080b13e" containerID="feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664" exitCode=0 Oct 30 02:47:35.266956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.266838 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" Oct 30 02:47:35.266956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.266844 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" event={"ID":"a6bab1e7-a639-4904-a00f-05e71080b13e","Type":"ContainerDied","Data":"feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664"} Oct 30 02:47:35.266956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.266950 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw" event={"ID":"a6bab1e7-a639-4904-a00f-05e71080b13e","Type":"ContainerDied","Data":"0a230a352c9fd6af3e049fdf3ce44921057144a7e91d891436497b89e5227b24"} Oct 30 02:47:35.267242 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.266973 5035 scope.go:122] "RemoveContainer" containerID="feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664" Oct 30 02:47:35.277799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.277776 5035 scope.go:122] "RemoveContainer" containerID="feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664" Oct 30 02:47:35.278121 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:35.278098 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664\": container with ID starting with feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664 not found: ID does not exist" containerID="feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664" Oct 30 02:47:35.278193 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.278131 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664"} err="failed to get container status \"feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664\": rpc error: code = NotFound desc = could not find container \"feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664\": container with ID starting with feb59f3daea627bf2c2f268ae98e2156138baa75b2cfb9caeb419ae3905ef664 not found: ID does not exist" Oct 30 02:47:35.317397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.317339 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw"] Oct 30 02:47:35.326254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.326077 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-9a1cd0-686646dd55-vfsfw"] Oct 30 02:47:35.642922 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.642888 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6bab1e7-a639-4904-a00f-05e71080b13e" path="/var/lib/kubelet/pods/a6bab1e7-a639-4904-a00f-05e71080b13e/volumes" Oct 30 02:47:35.831958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.831905 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm"] Oct 30 02:47:35.832168 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:35.832144 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" containerID="cri-o://38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487" gracePeriod=30 Oct 30 02:47:36.275088 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:36.275046 5035 generic.go:356] "Generic (PLEG): container finished" podID="3d626614-4ffd-4020-a2c6-987a96bd21dd" containerID="926e9e3ad2eefec02e12409a178b95c50b5c76bd73e90e8065c7e71d3c2ca1ba" exitCode=0 Oct 30 02:47:36.275579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:36.275135 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" event={"ID":"3d626614-4ffd-4020-a2c6-987a96bd21dd","Type":"ContainerDied","Data":"926e9e3ad2eefec02e12409a178b95c50b5c76bd73e90e8065c7e71d3c2ca1ba"} Oct 30 02:47:37.427771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.427683 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:37.464056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.464022 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-openshift-ca\") pod \"3d626614-4ffd-4020-a2c6-987a96bd21dd\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " Oct 30 02:47:37.464246 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.464075 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-xks-ca\") pod \"3d626614-4ffd-4020-a2c6-987a96bd21dd\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " Oct 30 02:47:37.464246 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.464169 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d626614-4ffd-4020-a2c6-987a96bd21dd-kube-api-access-qpbpq\" (UniqueName: \"kubernetes.io/projected/3d626614-4ffd-4020-a2c6-987a96bd21dd-kube-api-access-qpbpq\") pod \"3d626614-4ffd-4020-a2c6-987a96bd21dd\" (UID: \"3d626614-4ffd-4020-a2c6-987a96bd21dd\") " Oct 30 02:47:37.464398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.464373 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-openshift-ca" pod "3d626614-4ffd-4020-a2c6-987a96bd21dd" (UID: "3d626614-4ffd-4020-a2c6-987a96bd21dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:37.464491 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.464477 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:37.466496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.466465 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d626614-4ffd-4020-a2c6-987a96bd21dd-kube-api-access-qpbpq" pod "3d626614-4ffd-4020-a2c6-987a96bd21dd" (UID: "3d626614-4ffd-4020-a2c6-987a96bd21dd"). InnerVolumeSpecName "kube-api-access-qpbpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:37.466651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.466464 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-xks-ca" pod "3d626614-4ffd-4020-a2c6-987a96bd21dd" (UID: "3d626614-4ffd-4020-a2c6-987a96bd21dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:37.565368 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.565282 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d626614-4ffd-4020-a2c6-987a96bd21dd-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:37.565368 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:37.565316 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpbpq\" (UniqueName: \"kubernetes.io/projected/3d626614-4ffd-4020-a2c6-987a96bd21dd-kube-api-access-qpbpq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:38.289213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:38.289173 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" Oct 30 02:47:38.289213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:38.289207 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt" event={"ID":"3d626614-4ffd-4020-a2c6-987a96bd21dd","Type":"ContainerDied","Data":"3ab62b9fe983ce0daf390c441ef843116b55a2e2aa26241b0b170a9d23d23203"} Oct 30 02:47:38.289417 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:38.289259 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ab62b9fe983ce0daf390c441ef843116b55a2e2aa26241b0b170a9d23d23203" Oct 30 02:47:38.882094 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:38.882021 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" podUID="605e1a69-357a-4cba-8a4a-b88c8bb5f146" containerName="payload-processing" containerID="cri-o://8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c" gracePeriod=25 Oct 30 02:47:38.911645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:38.911580 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" podUID="d07d3256-13c9-4f87-a5b1-7d50bbaed683" containerName="payload-pre-processing" containerID="cri-o://49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689" gracePeriod=25 Oct 30 02:47:39.836757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:39.836713 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" podUID="d07d3256-13c9-4f87-a5b1-7d50bbaed683" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:40.843241 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:40.843195 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" podUID="605e1a69-357a-4cba-8a4a-b88c8bb5f146" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:41.121333 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.121297 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:47:41.207859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.207818 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-data\") pod \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " Oct 30 02:47:41.208058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.207873 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-token\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-token\") pod \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " Oct 30 02:47:41.208058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.207930 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-envoy\") pod \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " Oct 30 02:47:41.208058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.207960 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istiod-ca-cert\") pod \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " Oct 30 02:47:41.208058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.207988 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-podinfo\") pod \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " Oct 30 02:47:41.208058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208019 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-socket\") pod \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " Oct 30 02:47:41.208058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208056 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-credential-socket\") pod \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " Oct 30 02:47:41.208496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208157 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-kube-api-access-q6b6p\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-kube-api-access-q6b6p\") pod \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " Oct 30 02:47:41.208496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208178 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-certs\") pod \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\" (UID: \"f26c7197-b43a-4196-a5c3-4cbb88c5754f\") " Oct 30 02:47:41.208496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208205 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-data" pod "f26c7197-b43a-4196-a5c3-4cbb88c5754f" (UID: "f26c7197-b43a-4196-a5c3-4cbb88c5754f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:41.208496 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208456 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-socket" pod "f26c7197-b43a-4196-a5c3-4cbb88c5754f" (UID: "f26c7197-b43a-4196-a5c3-4cbb88c5754f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:41.208731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208676 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.208731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208697 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.208919 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208896 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-certs" pod "f26c7197-b43a-4196-a5c3-4cbb88c5754f" (UID: "f26c7197-b43a-4196-a5c3-4cbb88c5754f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:41.209014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.208982 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-credential-socket" pod "f26c7197-b43a-4196-a5c3-4cbb88c5754f" (UID: "f26c7197-b43a-4196-a5c3-4cbb88c5754f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:41.209185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.209146 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istiod-ca-cert" pod "f26c7197-b43a-4196-a5c3-4cbb88c5754f" (UID: "f26c7197-b43a-4196-a5c3-4cbb88c5754f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:41.210627 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.210594 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-envoy" pod "f26c7197-b43a-4196-a5c3-4cbb88c5754f" (UID: "f26c7197-b43a-4196-a5c3-4cbb88c5754f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:41.210981 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.210958 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-kube-api-access-q6b6p" pod "f26c7197-b43a-4196-a5c3-4cbb88c5754f" (UID: "f26c7197-b43a-4196-a5c3-4cbb88c5754f"). InnerVolumeSpecName "kube-api-access-q6b6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:41.211210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.211183 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-token" pod "f26c7197-b43a-4196-a5c3-4cbb88c5754f" (UID: "f26c7197-b43a-4196-a5c3-4cbb88c5754f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:41.211558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.211506 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-podinfo" pod "f26c7197-b43a-4196-a5c3-4cbb88c5754f" (UID: "f26c7197-b43a-4196-a5c3-4cbb88c5754f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:47:41.309391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.309359 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.309391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.309388 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.309651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.309401 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.309651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.309416 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6b6p\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-kube-api-access-q6b6p\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.309651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.309427 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.309651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.309439 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.309651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.309449 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26c7197-b43a-4196-a5c3-4cbb88c5754f-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:41.316199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.316158 5035 generic.go:356] "Generic (PLEG): container finished" podID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerID="38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487" exitCode=0 Oct 30 02:47:41.316422 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.316312 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" event={"ID":"f26c7197-b43a-4196-a5c3-4cbb88c5754f","Type":"ContainerDied","Data":"38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487"} Oct 30 02:47:41.316422 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.316364 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" event={"ID":"f26c7197-b43a-4196-a5c3-4cbb88c5754f","Type":"ContainerDied","Data":"91ea7651d24297e1347fa90a8f7a005630ea7e4a45dde444add335126552fd51"} Oct 30 02:47:41.316422 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.316392 5035 scope.go:122] "RemoveContainer" containerID="38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487" Oct 30 02:47:41.316655 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.316637 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm" Oct 30 02:47:41.329766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.329741 5035 scope.go:122] "RemoveContainer" containerID="38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487" Oct 30 02:47:41.330109 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:41.330090 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487\": container with ID starting with 38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487 not found: ID does not exist" containerID="38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487" Oct 30 02:47:41.330182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.330126 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487"} err="failed to get container status \"38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487\": rpc error: code = NotFound desc = could not find container \"38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487\": container with ID starting with 38349dc9b69cba047db2539745b926b04a26506fe5f9cad63d37d039b1999487 not found: ID does not exist" Oct 30 02:47:41.371398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.371296 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm"] Oct 30 02:47:41.379226 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.379192 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5htrm"] Oct 30 02:47:41.642782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:41.642696 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" path="/var/lib/kubelet/pods/f26c7197-b43a-4196-a5c3-4cbb88c5754f/volumes" Oct 30 02:47:42.193075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.193009 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" containerID="cri-o://865f4e8defcda3400581281601bb9de915e145ae474717286a46ab0a9e70e5e3" gracePeriod=45 Oct 30 02:47:42.216913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.216874 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Oct 30 02:47:42.326300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.326263 5035 generic.go:356] "Generic (PLEG): container finished" podID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerID="865f4e8defcda3400581281601bb9de915e145ae474717286a46ab0a9e70e5e3" exitCode=0 Oct 30 02:47:42.326484 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.326315 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" event={"ID":"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0","Type":"ContainerDied","Data":"865f4e8defcda3400581281601bb9de915e145ae474717286a46ab0a9e70e5e3"} Oct 30 02:47:42.491358 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.491327 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 02:47:42.524153 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.524115 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-model-cache\") pod \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " Oct 30 02:47:42.524391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.524201 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-kube-api-access-njgtq\" (UniqueName: \"kubernetes.io/projected/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-kube-api-access-njgtq\") pod \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " Oct 30 02:47:42.524391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.524277 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tmp-dir\") pod \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " Oct 30 02:47:42.524391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.524332 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tls-certs\") pod \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " Oct 30 02:47:42.524703 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.524404 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-home\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-home\") pod \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " Oct 30 02:47:42.524703 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.524445 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-dshm\") pod \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\" (UID: \"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0\") " Oct 30 02:47:42.524965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.524935 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-home" pod "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" (UID: "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.525087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.525061 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-model-cache" pod "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" (UID: "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.525187 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.525090 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tmp-dir" pod "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" (UID: "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.525433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.525395 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.525433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.525429 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.525625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.525444 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.527152 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.527122 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-kube-api-access-njgtq" pod "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" (UID: "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0"). InnerVolumeSpecName "kube-api-access-njgtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:42.527558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.527408 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-dshm" pod "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" (UID: "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.527558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.527505 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tls-certs" pod "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" (UID: "1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:42.626465 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.626421 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njgtq\" (UniqueName: \"kubernetes.io/projected/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-kube-api-access-njgtq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.626465 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.626468 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.626714 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.626483 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.776308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.776217 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" podUID="1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:42.776486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:42.776236 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" podUID="00e5e32e-4e1a-4585-967c-d98ad0617ec9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:43.338057 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:43.338019 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" Oct 30 02:47:43.338585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:43.338056 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns" event={"ID":"1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0","Type":"ContainerDied","Data":"7ee5577868e9e8c03c93b9c0793535d3d643b3963c8e560b0df0a79ef284c8f9"} Oct 30 02:47:43.338585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:43.338116 5035 scope.go:122] "RemoveContainer" containerID="865f4e8defcda3400581281601bb9de915e145ae474717286a46ab0a9e70e5e3" Oct 30 02:47:43.391053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:43.391015 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns"] Oct 30 02:47:43.403086 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:43.403053 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/distinct-2-w0-c40a89-kserve-754ffb9d57-gn7ns"] Oct 30 02:47:43.642183 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:43.642100 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" path="/var/lib/kubelet/pods/1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0/volumes" Oct 30 02:47:44.783687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:44.783654 5035 scope.go:122] "RemoveContainer" containerID="bec2352d71cf6d8b40f480c2369fa6cc62342d49053711cf6e4147cf13ed0209" Oct 30 02:47:47.629134 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:47.629092 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:47:49.837416 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:49.837371 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" podUID="d07d3256-13c9-4f87-a5b1-7d50bbaed683" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:50.844771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:50.844729 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" podUID="605e1a69-357a-4cba-8a4a-b88c8bb5f146" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:47:51.938763 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:51.938737 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 02:47:51.966260 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:51.966227 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 02:47:52.032470 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.032436 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e5e32e-4e1a-4585-967c-d98ad0617ec9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e5e32e-4e1a-4585-967c-d98ad0617ec9-plugins-config-volume\") pod \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\" (UID: \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\") " Oct 30 02:47:52.032670 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.032649 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-plugins-config-volume\") pod \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\" (UID: \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\") " Oct 30 02:47:52.032721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.032704 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-kube-api-access-wrxq5\" (UniqueName: \"kubernetes.io/projected/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-kube-api-access-wrxq5\") pod \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\" (UID: \"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636\") " Oct 30 02:47:52.032779 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.032759 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e5e32e-4e1a-4585-967c-d98ad0617ec9-kube-api-access-km4ww\" (UniqueName: \"kubernetes.io/projected/00e5e32e-4e1a-4585-967c-d98ad0617ec9-kube-api-access-km4ww\") pod \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\" (UID: \"00e5e32e-4e1a-4585-967c-d98ad0617ec9\") " Oct 30 02:47:52.033034 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.033010 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e5e32e-4e1a-4585-967c-d98ad0617ec9-plugins-config-volume" pod "00e5e32e-4e1a-4585-967c-d98ad0617ec9" (UID: "00e5e32e-4e1a-4585-967c-d98ad0617ec9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:52.033234 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.033059 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-plugins-config-volume" pod "1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" (UID: "1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:52.035021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.034989 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e5e32e-4e1a-4585-967c-d98ad0617ec9-kube-api-access-km4ww" pod "00e5e32e-4e1a-4585-967c-d98ad0617ec9" (UID: "00e5e32e-4e1a-4585-967c-d98ad0617ec9"). InnerVolumeSpecName "kube-api-access-km4ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:52.035157 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.035133 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-kube-api-access-wrxq5" pod "1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" (UID: "1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636"). InnerVolumeSpecName "kube-api-access-wrxq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:52.134408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.134359 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4ww\" (UniqueName: \"kubernetes.io/projected/00e5e32e-4e1a-4585-967c-d98ad0617ec9-kube-api-access-km4ww\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:52.134408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.134399 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e5e32e-4e1a-4585-967c-d98ad0617ec9-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:52.134408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.134413 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:52.134722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.134424 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrxq5\" (UniqueName: \"kubernetes.io/projected/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636-kube-api-access-wrxq5\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:52.406568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.406407 5035 generic.go:356] "Generic (PLEG): container finished" podID="00e5e32e-4e1a-4585-967c-d98ad0617ec9" containerID="dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548" exitCode=0 Oct 30 02:47:52.407221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.406516 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" event={"ID":"00e5e32e-4e1a-4585-967c-d98ad0617ec9","Type":"ContainerDied","Data":"dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548"} Oct 30 02:47:52.407221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.406603 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" Oct 30 02:47:52.407221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.406734 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6" event={"ID":"00e5e32e-4e1a-4585-967c-d98ad0617ec9","Type":"ContainerDied","Data":"4a2ced410e66433dd09dde075228b6d9ff7cd24a5790b5922d37cfd590bc1381"} Oct 30 02:47:52.407221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.406789 5035 scope.go:122] "RemoveContainer" containerID="dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548" Oct 30 02:47:52.410795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.410706 5035 generic.go:356] "Generic (PLEG): container finished" podID="1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" containerID="ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d" exitCode=0 Oct 30 02:47:52.411014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.410820 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" Oct 30 02:47:52.411014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.410823 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" event={"ID":"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636","Type":"ContainerDied","Data":"ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d"} Oct 30 02:47:52.411014 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.410962 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6" event={"ID":"1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636","Type":"ContainerDied","Data":"d665872ad1da5b6995fe1f3cc908ddc2d4f43a9500d97f4d01d4ae5aac74edab"} Oct 30 02:47:52.423869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.423829 5035 scope.go:122] "RemoveContainer" containerID="dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548" Oct 30 02:47:52.424323 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:52.424294 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548\": container with ID starting with dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548 not found: ID does not exist" containerID="dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548" Oct 30 02:47:52.424406 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.424338 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548"} err="failed to get container status \"dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548\": rpc error: code = NotFound desc = could not find container \"dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548\": container with ID starting with dc40851e46a88e10c4e604aec5cf884862a2caf185f29ebfa3f13bf3655a6548 not found: ID does not exist" Oct 30 02:47:52.424406 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.424366 5035 scope.go:122] "RemoveContainer" containerID="ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d" Oct 30 02:47:52.436690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.436654 5035 scope.go:122] "RemoveContainer" containerID="ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d" Oct 30 02:47:52.437060 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:52.437028 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d\": container with ID starting with ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d not found: ID does not exist" containerID="ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d" Oct 30 02:47:52.437163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.437070 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d"} err="failed to get container status \"ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d\": rpc error: code = NotFound desc = could not find container \"ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d\": container with ID starting with ffccfd8967194da36b04813e32c3b418e094c9abd98c65a4878f9c6e3e0e668d not found: ID does not exist" Oct 30 02:47:52.474062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.474012 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6"] Oct 30 02:47:52.477795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.477743 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-78d5ff-7c6d864484-n6nd6"] Oct 30 02:47:52.513289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.513216 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6"] Oct 30 02:47:52.516904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:52.516873 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-78d5ff-776dc9868c-wpbv6"] Oct 30 02:47:53.642159 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:53.642122 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e5e32e-4e1a-4585-967c-d98ad0617ec9" path="/var/lib/kubelet/pods/00e5e32e-4e1a-4585-967c-d98ad0617ec9/volumes" Oct 30 02:47:53.642608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:53.642519 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" path="/var/lib/kubelet/pods/1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636/volumes" Oct 30 02:47:54.288574 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:54.288510 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-45k2d_e33eacd4-92b9-47b7-b441-7f0b47bd6071/manager/0.log" Oct 30 02:47:54.710728 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:54.710695 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8fhqf_6e497615-20c6-4568-a41b-f8220a1f2e6c/kuadrant-console-plugin/0.log" Oct 30 02:47:55.595806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:55.595748 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-57dvj_adaf904e-ceb8-48d6-816f-3316258fe072/manager/0.log" Oct 30 02:47:59.165076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.165043 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 02:47:59.199801 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.199765 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 02:47:59.318278 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.318234 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07d3256-13c9-4f87-a5b1-7d50bbaed683-kube-api-access-6r89z\" (UniqueName: \"kubernetes.io/projected/d07d3256-13c9-4f87-a5b1-7d50bbaed683-kube-api-access-6r89z\") pod \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\" (UID: \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\") " Oct 30 02:47:59.318469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.318300 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07d3256-13c9-4f87-a5b1-7d50bbaed683-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d07d3256-13c9-4f87-a5b1-7d50bbaed683-plugins-config-volume\") pod \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\" (UID: \"d07d3256-13c9-4f87-a5b1-7d50bbaed683\") " Oct 30 02:47:59.318469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.318450 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605e1a69-357a-4cba-8a4a-b88c8bb5f146-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605e1a69-357a-4cba-8a4a-b88c8bb5f146-plugins-config-volume\") pod \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\" (UID: \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\") " Oct 30 02:47:59.318595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.318488 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605e1a69-357a-4cba-8a4a-b88c8bb5f146-kube-api-access-rwhlq\" (UniqueName: \"kubernetes.io/projected/605e1a69-357a-4cba-8a4a-b88c8bb5f146-kube-api-access-rwhlq\") pod \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\" (UID: \"605e1a69-357a-4cba-8a4a-b88c8bb5f146\") " Oct 30 02:47:59.318799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.318765 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07d3256-13c9-4f87-a5b1-7d50bbaed683-plugins-config-volume" pod "d07d3256-13c9-4f87-a5b1-7d50bbaed683" (UID: "d07d3256-13c9-4f87-a5b1-7d50bbaed683"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:59.318942 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.318901 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d07d3256-13c9-4f87-a5b1-7d50bbaed683-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:59.318987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.318928 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605e1a69-357a-4cba-8a4a-b88c8bb5f146-plugins-config-volume" pod "605e1a69-357a-4cba-8a4a-b88c8bb5f146" (UID: "605e1a69-357a-4cba-8a4a-b88c8bb5f146"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:59.320757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.320724 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07d3256-13c9-4f87-a5b1-7d50bbaed683-kube-api-access-6r89z" pod "d07d3256-13c9-4f87-a5b1-7d50bbaed683" (UID: "d07d3256-13c9-4f87-a5b1-7d50bbaed683"). InnerVolumeSpecName "kube-api-access-6r89z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:59.320875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.320851 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605e1a69-357a-4cba-8a4a-b88c8bb5f146-kube-api-access-rwhlq" pod "605e1a69-357a-4cba-8a4a-b88c8bb5f146" (UID: "605e1a69-357a-4cba-8a4a-b88c8bb5f146"). InnerVolumeSpecName "kube-api-access-rwhlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:59.420265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.420170 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/605e1a69-357a-4cba-8a4a-b88c8bb5f146-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:59.420265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.420204 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwhlq\" (UniqueName: \"kubernetes.io/projected/605e1a69-357a-4cba-8a4a-b88c8bb5f146-kube-api-access-rwhlq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:59.420265 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.420214 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r89z\" (UniqueName: \"kubernetes.io/projected/d07d3256-13c9-4f87-a5b1-7d50bbaed683-kube-api-access-6r89z\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:47:59.458874 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.458832 5035 generic.go:356] "Generic (PLEG): container finished" podID="605e1a69-357a-4cba-8a4a-b88c8bb5f146" containerID="8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c" exitCode=0 Oct 30 02:47:59.459070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.458908 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" Oct 30 02:47:59.459070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.458912 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" event={"ID":"605e1a69-357a-4cba-8a4a-b88c8bb5f146","Type":"ContainerDied","Data":"8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c"} Oct 30 02:47:59.459070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.458999 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6" event={"ID":"605e1a69-357a-4cba-8a4a-b88c8bb5f146","Type":"ContainerDied","Data":"a1fd566d70b4253fc06447e78ba582f80a33b96fb8854748466aa0bacc798cab"} Oct 30 02:47:59.459070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.459023 5035 scope.go:122] "RemoveContainer" containerID="8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c" Oct 30 02:47:59.460546 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.460500 5035 generic.go:356] "Generic (PLEG): container finished" podID="d07d3256-13c9-4f87-a5b1-7d50bbaed683" containerID="49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689" exitCode=0 Oct 30 02:47:59.460687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.460556 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" Oct 30 02:47:59.460687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.460554 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" event={"ID":"d07d3256-13c9-4f87-a5b1-7d50bbaed683","Type":"ContainerDied","Data":"49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689"} Oct 30 02:47:59.460687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.460601 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl" event={"ID":"d07d3256-13c9-4f87-a5b1-7d50bbaed683","Type":"ContainerDied","Data":"8a9e107557a64ecd005601e974bab26686827d4aa3c1a924811665f5cadf959b"} Oct 30 02:47:59.474055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.473397 5035 scope.go:122] "RemoveContainer" containerID="8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c" Oct 30 02:47:59.475705 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:59.475506 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c\": container with ID starting with 8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c not found: ID does not exist" containerID="8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c" Oct 30 02:47:59.475705 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.475703 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c"} err="failed to get container status \"8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c\": rpc error: code = NotFound desc = could not find container \"8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c\": container with ID starting with 8a8fa6f028a42db951097253b50074374e294f8f738080d3315b96d23f6c069c not found: ID does not exist" Oct 30 02:47:59.476173 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.475750 5035 scope.go:122] "RemoveContainer" containerID="49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689" Oct 30 02:47:59.486182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.486157 5035 scope.go:122] "RemoveContainer" containerID="49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689" Oct 30 02:47:59.486543 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:47:59.486505 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689\": container with ID starting with 49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689 not found: ID does not exist" containerID="49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689" Oct 30 02:47:59.486617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.486553 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689"} err="failed to get container status \"49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689\": rpc error: code = NotFound desc = could not find container \"49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689\": container with ID starting with 49159c0ffb75be3a5a7207518983b40a1315e6cae29d9a69e1be578e6ec21689 not found: ID does not exist" Oct 30 02:47:59.511759 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.511709 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl"] Oct 30 02:47:59.514146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.514107 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c40a89-5bcc67755f-wsrrl"] Oct 30 02:47:59.546894 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.546854 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6"] Oct 30 02:47:59.551159 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.551129 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c40a89-6759b44d8-tz9b6"] Oct 30 02:47:59.644666 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.644621 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605e1a69-357a-4cba-8a4a-b88c8bb5f146" path="/var/lib/kubelet/pods/605e1a69-357a-4cba-8a4a-b88c8bb5f146/volumes" Oct 30 02:47:59.645068 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:47:59.645055 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07d3256-13c9-4f87-a5b1-7d50bbaed683" path="/var/lib/kubelet/pods/d07d3256-13c9-4f87-a5b1-7d50bbaed683/volumes" Oct 30 02:48:03.608711 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.608674 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf"] Oct 30 02:48:03.609146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609124 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d07d3256-13c9-4f87-a5b1-7d50bbaed683" containerName="payload-pre-processing" Oct 30 02:48:03.609146 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609137 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07d3256-13c9-4f87-a5b1-7d50bbaed683" containerName="payload-pre-processing" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609152 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609158 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609167 5035 cpu_manager.go:414] "containerMap: removing container" podUID="00e5e32e-4e1a-4585-967c-d98ad0617ec9" containerName="payload-pre-processing" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609172 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e5e32e-4e1a-4585-967c-d98ad0617ec9" containerName="payload-pre-processing" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609198 5035 cpu_manager.go:414] "containerMap: removing container" podUID="3d626614-4ffd-4020-a2c6-987a96bd21dd" containerName="revoke-keys" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609204 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d626614-4ffd-4020-a2c6-987a96bd21dd" containerName="revoke-keys" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609216 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609221 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609228 5035 cpu_manager.go:414] "containerMap: removing container" podUID="a6bab1e7-a639-4904-a00f-05e71080b13e" containerName="payload-pre-processing" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609232 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6bab1e7-a639-4904-a00f-05e71080b13e" containerName="payload-pre-processing" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609240 5035 cpu_manager.go:414] "containerMap: removing container" podUID="1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" containerName="payload-processing" Oct 30 02:48:03.609240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609244 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" containerName="payload-processing" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609251 5035 cpu_manager.go:414] "containerMap: removing container" podUID="605e1a69-357a-4cba-8a4a-b88c8bb5f146" containerName="payload-processing" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609258 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605e1a69-357a-4cba-8a4a-b88c8bb5f146" containerName="payload-processing" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609332 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26c7197-b43a-4196-a5c3-4cbb88c5754f" containerName="istio-proxy" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609347 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e5e32e-4e1a-4585-967c-d98ad0617ec9" containerName="payload-pre-processing" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609356 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d626614-4ffd-4020-a2c6-987a96bd21dd" containerName="revoke-keys" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609365 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab24a6c-d9c8-4c42-b29b-24a7e2a7bfe0" containerName="main" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609370 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6bab1e7-a639-4904-a00f-05e71080b13e" containerName="payload-pre-processing" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609375 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5d3bb8-7e46-48c0-bb2c-ffd5e7337636" containerName="payload-processing" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609381 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="605e1a69-357a-4cba-8a4a-b88c8bb5f146" containerName="payload-processing" Oct 30 02:48:03.609671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.609389 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07d3256-13c9-4f87-a5b1-7d50bbaed683" containerName="payload-pre-processing" Oct 30 02:48:03.614000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.613978 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.642451 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.642421 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a0944980-openshift-default-dockercfg-r98r6\"" Oct 30 02:48:03.663718 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.663683 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf"] Oct 30 02:48:03.770247 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.770153 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8fc8\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-kube-api-access-m8fc8\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.770690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.770353 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-podinfo\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.770690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.770487 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-credential-socket\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.770690 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.770633 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-data\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.770870 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.770734 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-token\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.770968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.770933 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-envoy\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.771036 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.770998 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-socket\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.771084 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.771037 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa1d9fe-b9d1-4368-8714-050d565785b6-istiod-ca-cert\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.771084 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.771070 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-certs\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872413 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-podinfo\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872462 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-credential-socket\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872565 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-data\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872635 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-token\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872674 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-envoy\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872712 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-socket\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872774 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872751 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa1d9fe-b9d1-4368-8714-050d565785b6-istiod-ca-cert\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872788 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-certs\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872863 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8fc8\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-kube-api-access-m8fc8\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.872966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.872903 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-credential-socket\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.873278 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.873257 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-certs\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.873556 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.873518 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-socket\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.873922 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.873800 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-data\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.873993 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.873976 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa1d9fe-b9d1-4368-8714-050d565785b6-istiod-ca-cert\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.875502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.875474 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-envoy\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.875704 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.875684 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-podinfo\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.896169 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.896128 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-token\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.896357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.896326 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8fc8\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-kube-api-access-m8fc8\") pod \"e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:03.927872 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:03.927732 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:04.264296 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:04.264266 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf"] Oct 30 02:48:04.266055 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:48:04.266023 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa1d9fe_b9d1_4368_8714_050d565785b6.slice/crio-78bbac2401b991172c73beb5115db6d74759baa5c108cd0fb10e93a30ee7cb63 WatchSource:0}: Error finding container 78bbac2401b991172c73beb5115db6d74759baa5c108cd0fb10e93a30ee7cb63: Status 404 returned error can't find the container with id 78bbac2401b991172c73beb5115db6d74759baa5c108cd0fb10e93a30ee7cb63 Oct 30 02:48:04.268205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:04.268168 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:48:04.268284 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:04.268238 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:48:04.268284 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:04.268270 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:48:04.497110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:04.497039 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" event={"ID":"5fa1d9fe-b9d1-4368-8714-050d565785b6","Type":"ContainerStarted","Data":"03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552"} Oct 30 02:48:04.497381 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:04.497148 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" event={"ID":"5fa1d9fe-b9d1-4368-8714-050d565785b6","Type":"ContainerStarted","Data":"78bbac2401b991172c73beb5115db6d74759baa5c108cd0fb10e93a30ee7cb63"} Oct 30 02:48:04.545279 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:04.545160 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" podStartSLOduration=1.5451382900000001 podStartE2EDuration="1.54513829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:04.543699134 +0000 UTC m=+4763.046631350" watchObservedRunningTime="2026-10-30 02:48:04.54513829 +0000 UTC m=+4763.048070513" Oct 30 02:48:04.928710 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:04.928606 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:05.934490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:05.934453 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:06.508427 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:06.508392 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:06.509458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:06.509435 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:08.016282 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.016243 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt"] Oct 30 02:48:08.020592 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.020564 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:08.041723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.041687 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt"] Oct 30 02:48:08.050005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.049969 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a0944980\"" Oct 30 02:48:08.050210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.050116 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a0944980-dockercfg-bbgjq\"" Oct 30 02:48:08.116965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.116929 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1d89af-5e19-4acc-8d00-70094f1bc04e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt\" (UID: \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:08.117185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.117065 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7lk9\" (UniqueName: \"kubernetes.io/projected/9c1d89af-5e19-4acc-8d00-70094f1bc04e-kube-api-access-z7lk9\") pod \"payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt\" (UID: \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:08.218331 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.218280 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lk9\" (UniqueName: \"kubernetes.io/projected/9c1d89af-5e19-4acc-8d00-70094f1bc04e-kube-api-access-z7lk9\") pod \"payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt\" (UID: \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:08.218557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.218350 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1d89af-5e19-4acc-8d00-70094f1bc04e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt\" (UID: \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:08.219101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.219080 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1d89af-5e19-4acc-8d00-70094f1bc04e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt\" (UID: \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:08.242277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.242241 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7lk9\" (UniqueName: \"kubernetes.io/projected/9c1d89af-5e19-4acc-8d00-70094f1bc04e-kube-api-access-z7lk9\") pod \"payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt\" (UID: \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:08.334626 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.334484 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:08.655518 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:08.655480 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt"] Oct 30 02:48:08.657100 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:48:08.657070 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1d89af_5e19_4acc_8d00_70094f1bc04e.slice/crio-402889769d524558988c9edd1b05887323325ce32b77554b4abb7f0c911f1c19 WatchSource:0}: Error finding container 402889769d524558988c9edd1b05887323325ce32b77554b4abb7f0c911f1c19: Status 404 returned error can't find the container with id 402889769d524558988c9edd1b05887323325ce32b77554b4abb7f0c911f1c19 Oct 30 02:48:09.536742 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:09.536690 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" event={"ID":"9c1d89af-5e19-4acc-8d00-70094f1bc04e","Type":"ContainerStarted","Data":"c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd"} Oct 30 02:48:09.537200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:09.536799 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:09.537200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:09.536826 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" event={"ID":"9c1d89af-5e19-4acc-8d00-70094f1bc04e","Type":"ContainerStarted","Data":"402889769d524558988c9edd1b05887323325ce32b77554b4abb7f0c911f1c19"} Oct 30 02:48:09.579073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:09.579009 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" podStartSLOduration=2.5789940319999998 podStartE2EDuration="2.578994032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:48:09.57723836 +0000 UTC m=+4768.080170580" watchObservedRunningTime="2026-10-30 02:48:09.578994032 +0000 UTC m=+4768.081926251" Oct 30 02:48:17.633845 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:48:17.633740 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:48:20.546208 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:20.546178 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:21.510155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:21.510108 5035 patch_prober.go:29] interesting pod/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 02:48:21.510402 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:21.510187 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 02:48:21.643245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:21.643209 5035 generic.go:356] "Generic (PLEG): container finished" podID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerID="03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552" exitCode=0 Oct 30 02:48:21.643677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:21.643280 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" event={"ID":"5fa1d9fe-b9d1-4368-8714-050d565785b6","Type":"ContainerDied","Data":"03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552"} Oct 30 02:48:21.643723 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:21.643709 5035 scope.go:122] "RemoveContainer" containerID="03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552" Oct 30 02:48:21.644982 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:21.644954 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:48:21.645064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:21.645026 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:48:21.645064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:21.645057 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:48:22.030356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.030207 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm"] Oct 30 02:48:22.036547 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.036442 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.053473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.053359 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:48:22.054303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.054272 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:48:22.054452 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.054383 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:48:22.058727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.058690 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm"] Oct 30 02:48:22.064713 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.064651 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:48:22.101666 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.101616 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zztf\" (UniqueName: \"kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.101666 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.101673 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.101933 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.101708 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.203272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.203223 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zztf\" (UniqueName: \"kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.203272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.203282 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.203617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.203316 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.204134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.204103 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.205860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.205839 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.231497 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:48:22.231459 5035 projected.go:196] Error preparing data for projected volume kube-api-access-7zztf for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 02:48:22.231710 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:48:22.231555 5035 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf podName:32dae4ed-5e34-45fd-a070-411dbd92bb3d nodeName:}" failed. No retries permitted until 2026-10-30 02:48:22.73151637 +0000 UTC m=+4781.234448576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7zztf" (UniqueName: "kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf") pod "maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" (UID: "32dae4ed-5e34-45fd-a070-411dbd92bb3d") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 02:48:22.651195 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.651156 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" event={"ID":"5fa1d9fe-b9d1-4368-8714-050d565785b6","Type":"ContainerStarted","Data":"580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb"} Oct 30 02:48:22.810730 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.810676 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zztf\" (UniqueName: \"kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.834766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.834725 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zztf\" (UniqueName: \"kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf\") pod \"maas-api-revoke-keys-e2e-ait-a0944980-d4wsm\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:22.928569 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.928427 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:22.928893 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.928859 5035 patch_prober.go:29] interesting pod/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 02:48:22.928950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.928921 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 02:48:22.958914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.958877 5035 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" secret="" err="secret \"maas-api-cleanup-dockercfg-gmz8c\" not found" Oct 30 02:48:22.959144 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:22.958943 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:23.296325 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:23.296262 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm"] Oct 30 02:48:23.299604 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:48:23.299571 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32dae4ed_5e34_45fd_a070_411dbd92bb3d.slice/crio-da1644eea42b382adafd50f91863ddb2b9078d93245fffe7c2587eab8a0d1c7f WatchSource:0}: Error finding container da1644eea42b382adafd50f91863ddb2b9078d93245fffe7c2587eab8a0d1c7f: Status 404 returned error can't find the container with id da1644eea42b382adafd50f91863ddb2b9078d93245fffe7c2587eab8a0d1c7f Oct 30 02:48:23.658051 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:23.657953 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" event={"ID":"32dae4ed-5e34-45fd-a070-411dbd92bb3d","Type":"ContainerStarted","Data":"da1644eea42b382adafd50f91863ddb2b9078d93245fffe7c2587eab8a0d1c7f"} Oct 30 02:48:23.928679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:23.928584 5035 patch_prober.go:29] interesting pod/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 02:48:23.928679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:23.928640 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 02:48:24.694637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:24.694341 5035 generic.go:356] "Generic (PLEG): container finished" podID="32dae4ed-5e34-45fd-a070-411dbd92bb3d" containerID="51b3316aa023f4aea10583199610a5d391cbdc0fa7f828b3e9f936921420ec21" exitCode=0 Oct 30 02:48:24.694637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:24.694457 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" event={"ID":"32dae4ed-5e34-45fd-a070-411dbd92bb3d","Type":"ContainerDied","Data":"51b3316aa023f4aea10583199610a5d391cbdc0fa7f828b3e9f936921420ec21"} Oct 30 02:48:24.928962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:24.928915 5035 patch_prober.go:29] interesting pod/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 02:48:24.929134 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:24.928977 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 02:48:25.851408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:25.851381 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:25.929214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:25.929169 5035 patch_prober.go:29] interesting pod/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 02:48:25.929440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:25.929242 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 02:48:25.947083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:25.947040 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf\" (UniqueName: \"kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf\") pod \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " Oct 30 02:48:25.947282 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:25.947147 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-xks-ca\") pod \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " Oct 30 02:48:25.947282 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:25.947211 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-openshift-ca\") pod \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\" (UID: \"32dae4ed-5e34-45fd-a070-411dbd92bb3d\") " Oct 30 02:48:25.947648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:25.947623 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-openshift-ca" pod "32dae4ed-5e34-45fd-a070-411dbd92bb3d" (UID: "32dae4ed-5e34-45fd-a070-411dbd92bb3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:48:25.949413 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:25.949388 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-xks-ca" pod "32dae4ed-5e34-45fd-a070-411dbd92bb3d" (UID: "32dae4ed-5e34-45fd-a070-411dbd92bb3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:48:25.949522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:25.949471 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf" pod "32dae4ed-5e34-45fd-a070-411dbd92bb3d" (UID: "32dae4ed-5e34-45fd-a070-411dbd92bb3d"). InnerVolumeSpecName "kube-api-access-7zztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:48:26.048074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:26.048041 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:26.048074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:26.048071 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32dae4ed-5e34-45fd-a070-411dbd92bb3d-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:26.048074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:26.048081 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zztf\" (UniqueName: \"kubernetes.io/projected/32dae4ed-5e34-45fd-a070-411dbd92bb3d-kube-api-access-7zztf\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:26.708971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:26.708939 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" Oct 30 02:48:26.709185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:26.708973 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm" event={"ID":"32dae4ed-5e34-45fd-a070-411dbd92bb3d","Type":"ContainerDied","Data":"da1644eea42b382adafd50f91863ddb2b9078d93245fffe7c2587eab8a0d1c7f"} Oct 30 02:48:26.709185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:26.709022 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da1644eea42b382adafd50f91863ddb2b9078d93245fffe7c2587eab8a0d1c7f" Oct 30 02:48:26.928776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:26.928736 5035 patch_prober.go:29] interesting pod/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 02:48:26.929249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:26.928796 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 02:48:27.985807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:27.985771 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:27.986670 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:27.986636 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:27.988092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:27.988069 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:32.482607 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:32.482565 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt"] Oct 30 02:48:33.871405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:33.871360 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm"] Oct 30 02:48:33.874763 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:33.874731 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0944980-d4wsm"] Oct 30 02:48:35.643142 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:35.643104 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32dae4ed-5e34-45fd-a070-411dbd92bb3d" path="/var/lib/kubelet/pods/32dae4ed-5e34-45fd-a070-411dbd92bb3d/volumes" Oct 30 02:48:37.483873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.483813 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" podUID="9c1d89af-5e19-4acc-8d00-70094f1bc04e" containerName="payload-pre-processing" containerID="cri-o://c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd" gracePeriod=25 Oct 30 02:48:37.739000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.738934 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:37.777717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.777678 5035 generic.go:356] "Generic (PLEG): container finished" podID="9c1d89af-5e19-4acc-8d00-70094f1bc04e" containerID="c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd" exitCode=0 Oct 30 02:48:37.777911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.777737 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" Oct 30 02:48:37.777911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.777735 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" event={"ID":"9c1d89af-5e19-4acc-8d00-70094f1bc04e","Type":"ContainerDied","Data":"c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd"} Oct 30 02:48:37.777911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.777862 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt" event={"ID":"9c1d89af-5e19-4acc-8d00-70094f1bc04e","Type":"ContainerDied","Data":"402889769d524558988c9edd1b05887323325ce32b77554b4abb7f0c911f1c19"} Oct 30 02:48:37.777911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.777902 5035 scope.go:122] "RemoveContainer" containerID="c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd" Oct 30 02:48:37.789685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.789646 5035 scope.go:122] "RemoveContainer" containerID="c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd" Oct 30 02:48:37.790045 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:48:37.790011 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd\": container with ID starting with c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd not found: ID does not exist" containerID="c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd" Oct 30 02:48:37.790163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.790073 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd"} err="failed to get container status \"c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd\": rpc error: code = NotFound desc = could not find container \"c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd\": container with ID starting with c6f3e27841ac89de056d07e000b578137d928e0c7438265dc11685b14ca849fd not found: ID does not exist" Oct 30 02:48:37.883984 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.883841 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1d89af-5e19-4acc-8d00-70094f1bc04e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1d89af-5e19-4acc-8d00-70094f1bc04e-plugins-config-volume\") pod \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\" (UID: \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\") " Oct 30 02:48:37.885040 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.884451 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1d89af-5e19-4acc-8d00-70094f1bc04e-kube-api-access-z7lk9\" (UniqueName: \"kubernetes.io/projected/9c1d89af-5e19-4acc-8d00-70094f1bc04e-kube-api-access-z7lk9\") pod \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\" (UID: \"9c1d89af-5e19-4acc-8d00-70094f1bc04e\") " Oct 30 02:48:37.885205 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.885103 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1d89af-5e19-4acc-8d00-70094f1bc04e-plugins-config-volume" pod "9c1d89af-5e19-4acc-8d00-70094f1bc04e" (UID: "9c1d89af-5e19-4acc-8d00-70094f1bc04e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:48:37.890085 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.890015 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1d89af-5e19-4acc-8d00-70094f1bc04e-kube-api-access-z7lk9" pod "9c1d89af-5e19-4acc-8d00-70094f1bc04e" (UID: "9c1d89af-5e19-4acc-8d00-70094f1bc04e"). InnerVolumeSpecName "kube-api-access-z7lk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:48:37.986128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.986078 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7lk9\" (UniqueName: \"kubernetes.io/projected/9c1d89af-5e19-4acc-8d00-70094f1bc04e-kube-api-access-z7lk9\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:37.986128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:37.986120 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1d89af-5e19-4acc-8d00-70094f1bc04e-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:38.124103 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:38.124047 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt"] Oct 30 02:48:38.127539 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:38.127492 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a0944980-67bd8c5fc9-llfjt"] Oct 30 02:48:38.308861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:38.308823 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf"] Oct 30 02:48:38.309131 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:38.309096 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" containerID="cri-o://580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb" gracePeriod=30 Oct 30 02:48:39.641942 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:39.641890 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1d89af-5e19-4acc-8d00-70094f1bc04e" path="/var/lib/kubelet/pods/9c1d89af-5e19-4acc-8d00-70094f1bc04e/volumes" Oct 30 02:48:42.990549 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:42.990465 5035 patch_prober.go:29] interesting pod/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 02:48:42.991405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:42.991349 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:48:43.648369 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.648342 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:43.757112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757060 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-data\") pod \"5fa1d9fe-b9d1-4368-8714-050d565785b6\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " Oct 30 02:48:43.757342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757132 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-envoy\") pod \"5fa1d9fe-b9d1-4368-8714-050d565785b6\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " Oct 30 02:48:43.757342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757162 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-socket\") pod \"5fa1d9fe-b9d1-4368-8714-050d565785b6\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " Oct 30 02:48:43.757342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757190 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-token\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-token\") pod \"5fa1d9fe-b9d1-4368-8714-050d565785b6\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " Oct 30 02:48:43.757342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757235 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa1d9fe-b9d1-4368-8714-050d565785b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa1d9fe-b9d1-4368-8714-050d565785b6-istiod-ca-cert\") pod \"5fa1d9fe-b9d1-4368-8714-050d565785b6\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " Oct 30 02:48:43.757342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757271 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-kube-api-access-m8fc8\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-kube-api-access-m8fc8\") pod \"5fa1d9fe-b9d1-4368-8714-050d565785b6\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " Oct 30 02:48:43.757799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757366 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-credential-socket\") pod \"5fa1d9fe-b9d1-4368-8714-050d565785b6\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " Oct 30 02:48:43.757799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757375 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-data" pod "5fa1d9fe-b9d1-4368-8714-050d565785b6" (UID: "5fa1d9fe-b9d1-4368-8714-050d565785b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:48:43.757799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757471 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-podinfo\") pod \"5fa1d9fe-b9d1-4368-8714-050d565785b6\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " Oct 30 02:48:43.757799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757512 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-certs\") pod \"5fa1d9fe-b9d1-4368-8714-050d565785b6\" (UID: \"5fa1d9fe-b9d1-4368-8714-050d565785b6\") " Oct 30 02:48:43.757799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757502 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-socket" pod "5fa1d9fe-b9d1-4368-8714-050d565785b6" (UID: "5fa1d9fe-b9d1-4368-8714-050d565785b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:48:43.757799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757617 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-credential-socket" pod "5fa1d9fe-b9d1-4368-8714-050d565785b6" (UID: "5fa1d9fe-b9d1-4368-8714-050d565785b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:48:43.757799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757725 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa1d9fe-b9d1-4368-8714-050d565785b6-istiod-ca-cert" pod "5fa1d9fe-b9d1-4368-8714-050d565785b6" (UID: "5fa1d9fe-b9d1-4368-8714-050d565785b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:48:43.758156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.757861 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-certs" pod "5fa1d9fe-b9d1-4368-8714-050d565785b6" (UID: "5fa1d9fe-b9d1-4368-8714-050d565785b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:48:43.758156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.758042 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.758156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.758063 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.758156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.758079 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.758156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.758093 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.758156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.758106 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa1d9fe-b9d1-4368-8714-050d565785b6-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.759644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.759621 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-kube-api-access-m8fc8" pod "5fa1d9fe-b9d1-4368-8714-050d565785b6" (UID: "5fa1d9fe-b9d1-4368-8714-050d565785b6"). InnerVolumeSpecName "kube-api-access-m8fc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:48:43.759644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.759622 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-token" pod "5fa1d9fe-b9d1-4368-8714-050d565785b6" (UID: "5fa1d9fe-b9d1-4368-8714-050d565785b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:48:43.759833 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.759815 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-envoy" pod "5fa1d9fe-b9d1-4368-8714-050d565785b6" (UID: "5fa1d9fe-b9d1-4368-8714-050d565785b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:48:43.759905 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.759890 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-podinfo" pod "5fa1d9fe-b9d1-4368-8714-050d565785b6" (UID: "5fa1d9fe-b9d1-4368-8714-050d565785b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:48:43.823217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.823167 5035 generic.go:356] "Generic (PLEG): container finished" podID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerID="580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb" exitCode=0 Oct 30 02:48:43.823454 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.823336 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" event={"ID":"5fa1d9fe-b9d1-4368-8714-050d565785b6","Type":"ContainerDied","Data":"580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb"} Oct 30 02:48:43.823454 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.823397 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" event={"ID":"5fa1d9fe-b9d1-4368-8714-050d565785b6","Type":"ContainerDied","Data":"78bbac2401b991172c73beb5115db6d74759baa5c108cd0fb10e93a30ee7cb63"} Oct 30 02:48:43.823454 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.823431 5035 scope.go:122] "RemoveContainer" containerID="580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb" Oct 30 02:48:43.823696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.823615 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf" Oct 30 02:48:43.839029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.839006 5035 scope.go:122] "RemoveContainer" containerID="03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552" Oct 30 02:48:43.848212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.848186 5035 scope.go:122] "RemoveContainer" containerID="580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb" Oct 30 02:48:43.848565 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:48:43.848507 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb\": container with ID starting with 580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb not found: ID does not exist" containerID="580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb" Oct 30 02:48:43.848683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.848577 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb"} err="failed to get container status \"580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb\": rpc error: code = NotFound desc = could not find container \"580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb\": container with ID starting with 580aeb4f6c42f7c16aaa541518b7a3e9a99739a93f2ee5b0129b100cb1e2d5cb not found: ID does not exist" Oct 30 02:48:43.848683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.848604 5035 scope.go:122] "RemoveContainer" containerID="03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552" Oct 30 02:48:43.848916 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:48:43.848895 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552\": container with ID starting with 03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552 not found: ID does not exist" containerID="03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552" Oct 30 02:48:43.848968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.848923 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552"} err="failed to get container status \"03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552\": rpc error: code = NotFound desc = could not find container \"03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552\": container with ID starting with 03b74d071aae720601f7ead8c815607329af5a591ef896630cbbfdfea311b552 not found: ID does not exist" Oct 30 02:48:43.859180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.859153 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.859180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.859180 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.859366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.859192 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.859366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.859201 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8fc8\" (UniqueName: \"kubernetes.io/projected/5fa1d9fe-b9d1-4368-8714-050d565785b6-kube-api-access-m8fc8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:48:43.882305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.882193 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf"] Oct 30 02:48:43.886966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:43.886930 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a0944980-openshift-default-d4b4774f5-vvnrf"] Oct 30 02:48:44.149384 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:44.149291 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:48:44.160084 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:44.160054 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:48:44.164601 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:44.164575 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:48:44.230431 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:44.230240 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:48:44.243601 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:44.243573 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:48:44.248101 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:44.248077 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:48:45.641803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:48:45.641759 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" path="/var/lib/kubelet/pods/5fa1d9fe-b9d1-4368-8714-050d565785b6/volumes" Oct 30 02:48:47.629741 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:48:47.629699 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:49:03.209776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.209684 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx"] Oct 30 02:49:03.210271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210161 5035 cpu_manager.go:414] "containerMap: removing container" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" Oct 30 02:49:03.210271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210176 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" Oct 30 02:49:03.210271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210185 5035 cpu_manager.go:414] "containerMap: removing container" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" Oct 30 02:49:03.210271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210190 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" Oct 30 02:49:03.210271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210200 5035 cpu_manager.go:414] "containerMap: removing container" podUID="9c1d89af-5e19-4acc-8d00-70094f1bc04e" containerName="payload-pre-processing" Oct 30 02:49:03.210271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210205 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1d89af-5e19-4acc-8d00-70094f1bc04e" containerName="payload-pre-processing" Oct 30 02:49:03.210271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210224 5035 cpu_manager.go:414] "containerMap: removing container" podUID="32dae4ed-5e34-45fd-a070-411dbd92bb3d" containerName="revoke-keys" Oct 30 02:49:03.210271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210230 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32dae4ed-5e34-45fd-a070-411dbd92bb3d" containerName="revoke-keys" Oct 30 02:49:03.210576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210317 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" Oct 30 02:49:03.210576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210326 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa1d9fe-b9d1-4368-8714-050d565785b6" containerName="istio-proxy" Oct 30 02:49:03.210576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210332 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1d89af-5e19-4acc-8d00-70094f1bc04e" containerName="payload-pre-processing" Oct 30 02:49:03.210576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.210341 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="32dae4ed-5e34-45fd-a070-411dbd92bb3d" containerName="revoke-keys" Oct 30 02:49:03.213479 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.213456 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.235696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.235659 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx"] Oct 30 02:49:03.241250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.241202 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:49:03.241250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.241206 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:49:03.241250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.241233 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:49:03.241593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.241291 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:49:03.362950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.362883 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lhsfx\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.363240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.363101 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lhsfx\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.363240 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.363152 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfrrb\" (UniqueName: \"kubernetes.io/projected/ae9c6804-a967-446b-ae0b-3030b03909b7-kube-api-access-cfrrb\") pod \"maas-api-revoke-sub-85237e45-lhsfx\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.464245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.464147 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lhsfx\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.464245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.464194 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfrrb\" (UniqueName: \"kubernetes.io/projected/ae9c6804-a967-446b-ae0b-3030b03909b7-kube-api-access-cfrrb\") pod \"maas-api-revoke-sub-85237e45-lhsfx\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.464466 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.464250 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lhsfx\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.464898 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.464860 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lhsfx\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.466791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.466766 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lhsfx\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.487876 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.487835 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfrrb\" (UniqueName: \"kubernetes.io/projected/ae9c6804-a967-446b-ae0b-3030b03909b7-kube-api-access-cfrrb\") pod \"maas-api-revoke-sub-85237e45-lhsfx\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.526691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.526651 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:03.833696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.833663 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx"] Oct 30 02:49:03.836039 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:49:03.836005 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9c6804_a967_446b_ae0b_3030b03909b7.slice/crio-67fa16dcec261a306cd8dfabb1651d1a85c9b8958154e68375280b78e7c79e84 WatchSource:0}: Error finding container 67fa16dcec261a306cd8dfabb1651d1a85c9b8958154e68375280b78e7c79e84: Status 404 returned error can't find the container with id 67fa16dcec261a306cd8dfabb1651d1a85c9b8958154e68375280b78e7c79e84 Oct 30 02:49:03.943522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:03.943479 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" event={"ID":"ae9c6804-a967-446b-ae0b-3030b03909b7","Type":"ContainerStarted","Data":"67fa16dcec261a306cd8dfabb1651d1a85c9b8958154e68375280b78e7c79e84"} Oct 30 02:49:04.951270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:04.951230 5035 generic.go:356] "Generic (PLEG): container finished" podID="ae9c6804-a967-446b-ae0b-3030b03909b7" containerID="9302c78bf97f89fc392628660f16f32f4dc36307c33e6bf7727aa39c1618f599" exitCode=0 Oct 30 02:49:04.951729 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:04.951312 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" event={"ID":"ae9c6804-a967-446b-ae0b-3030b03909b7","Type":"ContainerDied","Data":"9302c78bf97f89fc392628660f16f32f4dc36307c33e6bf7727aa39c1618f599"} Oct 30 02:49:06.121685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.121652 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:06.123058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.123027 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw"] Oct 30 02:49:06.123523 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.123503 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ae9c6804-a967-446b-ae0b-3030b03909b7" containerName="revoke-keys" Oct 30 02:49:06.123634 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.123541 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9c6804-a967-446b-ae0b-3030b03909b7" containerName="revoke-keys" Oct 30 02:49:06.123699 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.123674 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9c6804-a967-446b-ae0b-3030b03909b7" containerName="revoke-keys" Oct 30 02:49:06.126875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.126847 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.159459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.159408 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw"] Oct 30 02:49:06.193178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.193082 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-xks-ca\") pod \"ae9c6804-a967-446b-ae0b-3030b03909b7\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " Oct 30 02:49:06.193564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.193281 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9c6804-a967-446b-ae0b-3030b03909b7-kube-api-access-cfrrb\" (UniqueName: \"kubernetes.io/projected/ae9c6804-a967-446b-ae0b-3030b03909b7-kube-api-access-cfrrb\") pod \"ae9c6804-a967-446b-ae0b-3030b03909b7\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " Oct 30 02:49:06.193564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.193353 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-openshift-ca\") pod \"ae9c6804-a967-446b-ae0b-3030b03909b7\" (UID: \"ae9c6804-a967-446b-ae0b-3030b03909b7\") " Oct 30 02:49:06.193740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.193606 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.193740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.193659 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.193740 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.193692 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.193910 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.193818 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.193910 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.193880 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-openshift-ca" pod "ae9c6804-a967-446b-ae0b-3030b03909b7" (UID: "ae9c6804-a967-446b-ae0b-3030b03909b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:49:06.194024 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.194002 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxb7\" (UniqueName: \"kubernetes.io/projected/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-kube-api-access-zkxb7\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.194093 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.194075 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.194199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.194170 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.194286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.194232 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.194405 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.194381 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.194573 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.194555 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:49:06.196502 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.196456 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-xks-ca" pod "ae9c6804-a967-446b-ae0b-3030b03909b7" (UID: "ae9c6804-a967-446b-ae0b-3030b03909b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:49:06.196693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.196465 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9c6804-a967-446b-ae0b-3030b03909b7-kube-api-access-cfrrb" pod "ae9c6804-a967-446b-ae0b-3030b03909b7" (UID: "ae9c6804-a967-446b-ae0b-3030b03909b7"). InnerVolumeSpecName "kube-api-access-cfrrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:49:06.297076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.296736 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.297076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.296924 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.297076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.297012 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.297781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.297111 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.297781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.297214 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.297781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.297308 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.297781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.297351 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.297781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.297395 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.297781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.297562 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxb7\" (UniqueName: \"kubernetes.io/projected/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-kube-api-access-zkxb7\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.297781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.297727 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae9c6804-a967-446b-ae0b-3030b03909b7-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:49:06.297781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.297758 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfrrb\" (UniqueName: \"kubernetes.io/projected/ae9c6804-a967-446b-ae0b-3030b03909b7-kube-api-access-cfrrb\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:49:06.298335 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.298098 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.298425 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.298382 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.298497 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.298398 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.298693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.298644 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.298805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.298632 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.301719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.301673 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.301901 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.301879 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.321490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.321445 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxb7\" (UniqueName: \"kubernetes.io/projected/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-kube-api-access-zkxb7\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.321741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.321445 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71091cb-5eac-44d9-8cba-bbc3f04d66d6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-zvgvw\" (UID: \"a71091cb-5eac-44d9-8cba-bbc3f04d66d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.440793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.440750 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:06.761934 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.761901 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw"] Oct 30 02:49:06.764478 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:49:06.764438 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71091cb_5eac_44d9_8cba_bbc3f04d66d6.slice/crio-19f8544dfc07a07ef7e909291cbf9132088fd2265fb62a71d15506aa7532ef3e WatchSource:0}: Error finding container 19f8544dfc07a07ef7e909291cbf9132088fd2265fb62a71d15506aa7532ef3e: Status 404 returned error can't find the container with id 19f8544dfc07a07ef7e909291cbf9132088fd2265fb62a71d15506aa7532ef3e Oct 30 02:49:06.766937 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.766898 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:49:06.767034 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.766999 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:49:06.767079 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.767031 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:49:06.965387 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.965343 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" event={"ID":"a71091cb-5eac-44d9-8cba-bbc3f04d66d6","Type":"ContainerStarted","Data":"f20a07886449e4a8494be3a5913ef55df63f0838b19a69a2356c83c7962428c0"} Oct 30 02:49:06.965613 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.965417 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" event={"ID":"a71091cb-5eac-44d9-8cba-bbc3f04d66d6","Type":"ContainerStarted","Data":"19f8544dfc07a07ef7e909291cbf9132088fd2265fb62a71d15506aa7532ef3e"} Oct 30 02:49:06.967219 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.967196 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" event={"ID":"ae9c6804-a967-446b-ae0b-3030b03909b7","Type":"ContainerDied","Data":"67fa16dcec261a306cd8dfabb1651d1a85c9b8958154e68375280b78e7c79e84"} Oct 30 02:49:06.967376 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.967220 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx" Oct 30 02:49:06.967376 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:06.967238 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67fa16dcec261a306cd8dfabb1651d1a85c9b8958154e68375280b78e7c79e84" Oct 30 02:49:07.012045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:07.011923 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podStartSLOduration=1.011900494 podStartE2EDuration="1.011900494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:07.010859604 +0000 UTC m=+4825.513791824" watchObservedRunningTime="2026-10-30 02:49:07.011900494 +0000 UTC m=+4825.514832713" Oct 30 02:49:07.441033 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:07.440936 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:07.442673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:07.442640 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 02:49:07.442838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:07.442703 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 02:49:08.441337 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:08.441296 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 02:49:08.441911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:08.441360 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 02:49:09.441401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:09.441353 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 02:49:09.441980 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:09.441429 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 02:49:10.442065 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:10.442024 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 02:49:10.442509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:10.442081 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 02:49:11.441298 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:11.441257 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 02:49:11.441510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:11.441317 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 02:49:12.442124 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:12.442081 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 02:49:12.442632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:12.442152 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 02:49:13.441688 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:13.441638 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 02:49:13.441896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:13.441716 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 02:49:14.442047 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:14.441978 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body= Oct 30 02:49:14.442623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:14.442081 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" Oct 30 02:49:15.445493 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:15.445457 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:15.445915 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:15.445782 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:15.446459 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:15.446435 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" Oct 30 02:49:17.629358 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:49:17.629314 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:49:21.165291 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.165245 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb"] Oct 30 02:49:21.170302 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.170278 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.201375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.201325 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb"] Oct 30 02:49:21.257030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.256982 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.257030 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.257033 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.257270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.257058 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807bae01-b730-4075-b8ec-eed162d6afcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.257270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.257200 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807bae01-b730-4075-b8ec-eed162d6afcd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.257270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.257261 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.257375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.257311 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wf2\" (UniqueName: \"kubernetes.io/projected/807bae01-b730-4075-b8ec-eed162d6afcd-kube-api-access-m4wf2\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.257375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.257344 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.257375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.257366 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.257481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.257386 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807bae01-b730-4075-b8ec-eed162d6afcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.358761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.358719 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.358978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.358797 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wf2\" (UniqueName: \"kubernetes.io/projected/807bae01-b730-4075-b8ec-eed162d6afcd-kube-api-access-m4wf2\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.358978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.358816 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.358978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.358837 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.358978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.358856 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807bae01-b730-4075-b8ec-eed162d6afcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.358978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.358890 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.358978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.358918 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.358978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.358952 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807bae01-b730-4075-b8ec-eed162d6afcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.359390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.358998 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807bae01-b730-4075-b8ec-eed162d6afcd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.359390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.359257 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.359390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.359257 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.359390 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.359324 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.359597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.359418 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.359897 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.359868 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807bae01-b730-4075-b8ec-eed162d6afcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.361404 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.361381 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807bae01-b730-4075-b8ec-eed162d6afcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.361541 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.361471 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807bae01-b730-4075-b8ec-eed162d6afcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.382418 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.382375 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807bae01-b730-4075-b8ec-eed162d6afcd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.382627 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.382463 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wf2\" (UniqueName: \"kubernetes.io/projected/807bae01-b730-4075-b8ec-eed162d6afcd-kube-api-access-m4wf2\") pod \"maas-default-gateway-openshift-default-75d7c785f-4spcb\" (UID: \"807bae01-b730-4075-b8ec-eed162d6afcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.486988 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.486937 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:21.812980 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.812946 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb"] Oct 30 02:49:21.815351 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:49:21.815318 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807bae01_b730_4075_b8ec_eed162d6afcd.slice/crio-bb7bc7d59b646db2908b185717b5f57f4ba4c013042223129b71a16311b2b4f4 WatchSource:0}: Error finding container bb7bc7d59b646db2908b185717b5f57f4ba4c013042223129b71a16311b2b4f4: Status 404 returned error can't find the container with id bb7bc7d59b646db2908b185717b5f57f4ba4c013042223129b71a16311b2b4f4 Oct 30 02:49:21.817593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.817556 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:49:21.817717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.817643 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:49:21.817717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:21.817690 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:49:22.088019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:22.087922 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" event={"ID":"807bae01-b730-4075-b8ec-eed162d6afcd","Type":"ContainerStarted","Data":"650a96c780ef47817a211665fa897294182c40a6c7366cebbbdf87b00be1119f"} Oct 30 02:49:22.088019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:22.087984 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" event={"ID":"807bae01-b730-4075-b8ec-eed162d6afcd","Type":"ContainerStarted","Data":"bb7bc7d59b646db2908b185717b5f57f4ba4c013042223129b71a16311b2b4f4"} Oct 30 02:49:22.135273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:22.135211 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podStartSLOduration=1.135196155 podStartE2EDuration="1.135196155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:22.134581673 +0000 UTC m=+4840.637513919" watchObservedRunningTime="2026-10-30 02:49:22.135196155 +0000 UTC m=+4840.638128404" Oct 30 02:49:22.487769 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:22.487726 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:22.490476 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:22.490439 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 02:49:22.490782 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:22.490745 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 02:49:23.491728 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:23.491664 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 02:49:23.492199 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:23.491768 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 02:49:24.492683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:24.492607 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 02:49:24.493500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:24.492720 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 02:49:25.488554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:25.488005 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 02:49:25.488554 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:25.488076 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 02:49:26.488058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:26.488012 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 02:49:26.488754 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:26.488715 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 02:49:27.490031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:27.489987 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 02:49:27.490710 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:27.490672 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 02:49:28.488143 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:28.488086 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 02:49:28.488351 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:28.488213 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 02:49:29.487382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:29.487342 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 02:49:29.487853 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:29.487416 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 02:49:30.492243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:30.492208 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:30.492797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:30.492638 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:30.493500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:30.493476 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" Oct 30 02:49:42.762495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.762453 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5"] Oct 30 02:49:42.766840 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.766813 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:42.783748 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.783704 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:49:42.795631 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.795591 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:49:42.795631 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.795616 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:49:42.795893 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.795647 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:49:42.798900 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.798863 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5"] Oct 30 02:49:42.876849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.876802 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-knvh5\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:42.876849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.876857 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96zrr\" (UniqueName: \"kubernetes.io/projected/49e026bb-879a-4fc2-b0b6-5322d024de95-kube-api-access-96zrr\") pod \"maas-api-revoke-sub-f904bff9-knvh5\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:42.877076 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.876989 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-knvh5\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:42.978324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.978285 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-knvh5\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:42.978566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.978338 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96zrr\" (UniqueName: \"kubernetes.io/projected/49e026bb-879a-4fc2-b0b6-5322d024de95-kube-api-access-96zrr\") pod \"maas-api-revoke-sub-f904bff9-knvh5\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:42.978566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.978419 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-knvh5\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:42.979070 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.979040 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-knvh5\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:42.980919 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:42.980890 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-knvh5\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:43.001806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:43.001775 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96zrr\" (UniqueName: \"kubernetes.io/projected/49e026bb-879a-4fc2-b0b6-5322d024de95-kube-api-access-96zrr\") pod \"maas-api-revoke-sub-f904bff9-knvh5\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:43.080919 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:43.080827 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:43.383587 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:43.383553 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5"] Oct 30 02:49:43.385608 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:49:43.385577 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e026bb_879a_4fc2_b0b6_5322d024de95.slice/crio-3314a03da2f8b3aa45908840b91e4afa839a094fd873a466657b4bc5161ed9f4 WatchSource:0}: Error finding container 3314a03da2f8b3aa45908840b91e4afa839a094fd873a466657b4bc5161ed9f4: Status 404 returned error can't find the container with id 3314a03da2f8b3aa45908840b91e4afa839a094fd873a466657b4bc5161ed9f4 Oct 30 02:49:44.291472 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:44.291339 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" event={"ID":"49e026bb-879a-4fc2-b0b6-5322d024de95","Type":"ContainerStarted","Data":"5e55c8c968b4bd3566222c126bb7c5c8d83d5d1b1c130ff6f34c0943c60c572e"} Oct 30 02:49:44.291472 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:44.291420 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" event={"ID":"49e026bb-879a-4fc2-b0b6-5322d024de95","Type":"ContainerStarted","Data":"3314a03da2f8b3aa45908840b91e4afa839a094fd873a466657b4bc5161ed9f4"} Oct 30 02:49:45.299808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.299768 5035 generic.go:356] "Generic (PLEG): container finished" podID="49e026bb-879a-4fc2-b0b6-5322d024de95" containerID="5e55c8c968b4bd3566222c126bb7c5c8d83d5d1b1c130ff6f34c0943c60c572e" exitCode=0 Oct 30 02:49:45.300271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.299857 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" event={"ID":"49e026bb-879a-4fc2-b0b6-5322d024de95","Type":"ContainerDied","Data":"5e55c8c968b4bd3566222c126bb7c5c8d83d5d1b1c130ff6f34c0943c60c572e"} Oct 30 02:49:45.459485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.459456 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:45.606581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.606463 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e026bb-879a-4fc2-b0b6-5322d024de95-kube-api-access-96zrr\" (UniqueName: \"kubernetes.io/projected/49e026bb-879a-4fc2-b0b6-5322d024de95-kube-api-access-96zrr\") pod \"49e026bb-879a-4fc2-b0b6-5322d024de95\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " Oct 30 02:49:45.606581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.606542 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-openshift-ca\") pod \"49e026bb-879a-4fc2-b0b6-5322d024de95\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " Oct 30 02:49:45.606818 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.606650 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-xks-ca\") pod \"49e026bb-879a-4fc2-b0b6-5322d024de95\" (UID: \"49e026bb-879a-4fc2-b0b6-5322d024de95\") " Oct 30 02:49:45.606946 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.606916 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-openshift-ca" pod "49e026bb-879a-4fc2-b0b6-5322d024de95" (UID: "49e026bb-879a-4fc2-b0b6-5322d024de95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:49:45.607029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.607006 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:49:45.608804 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.608776 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e026bb-879a-4fc2-b0b6-5322d024de95-kube-api-access-96zrr" pod "49e026bb-879a-4fc2-b0b6-5322d024de95" (UID: "49e026bb-879a-4fc2-b0b6-5322d024de95"). InnerVolumeSpecName "kube-api-access-96zrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:49:45.608804 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.608797 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-xks-ca" pod "49e026bb-879a-4fc2-b0b6-5322d024de95" (UID: "49e026bb-879a-4fc2-b0b6-5322d024de95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:49:45.708693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.708646 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96zrr\" (UniqueName: \"kubernetes.io/projected/49e026bb-879a-4fc2-b0b6-5322d024de95-kube-api-access-96zrr\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:49:45.708693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:45.708687 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e026bb-879a-4fc2-b0b6-5322d024de95-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:49:46.307719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:46.307680 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" Oct 30 02:49:46.307719 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:46.307707 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5" event={"ID":"49e026bb-879a-4fc2-b0b6-5322d024de95","Type":"ContainerDied","Data":"3314a03da2f8b3aa45908840b91e4afa839a094fd873a466657b4bc5161ed9f4"} Oct 30 02:49:46.308222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:49:46.307761 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3314a03da2f8b3aa45908840b91e4afa839a094fd873a466657b4bc5161ed9f4" Oct 30 02:49:47.629220 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:49:47.629180 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:50:07.423674 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.423623 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8"] Oct 30 02:50:07.424231 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.424211 5035 cpu_manager.go:414] "containerMap: removing container" podUID="49e026bb-879a-4fc2-b0b6-5322d024de95" containerName="revoke-keys" Oct 30 02:50:07.424278 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.424232 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e026bb-879a-4fc2-b0b6-5322d024de95" containerName="revoke-keys" Oct 30 02:50:07.424328 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.424318 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e026bb-879a-4fc2-b0b6-5322d024de95" containerName="revoke-keys" Oct 30 02:50:07.427640 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.427619 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.456825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.456787 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:50:07.456825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.456803 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:50:07.457108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.456875 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:50:07.457108 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.456935 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:50:07.459509 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.459479 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8"] Oct 30 02:50:07.525475 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.525433 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxpn\" (UniqueName: \"kubernetes.io/projected/f6737d10-6c9f-4388-846f-85ee2f96e21c-kube-api-access-8fxpn\") pod \"maas-api-revoke-sub-f03629db-4f4c8\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.525709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.525604 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4f4c8\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.525709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.525636 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4f4c8\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.626985 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.626905 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxpn\" (UniqueName: \"kubernetes.io/projected/f6737d10-6c9f-4388-846f-85ee2f96e21c-kube-api-access-8fxpn\") pod \"maas-api-revoke-sub-f03629db-4f4c8\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.627271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.627080 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4f4c8\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.627271 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.627104 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4f4c8\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.628122 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.628054 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4f4c8\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.631409 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.631355 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4f4c8\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.650954 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.650917 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxpn\" (UniqueName: \"kubernetes.io/projected/f6737d10-6c9f-4388-846f-85ee2f96e21c-kube-api-access-8fxpn\") pod \"maas-api-revoke-sub-f03629db-4f4c8\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:07.742027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:07.741988 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:08.052970 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:08.052938 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8"] Oct 30 02:50:08.054851 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:50:08.054823 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6737d10_6c9f_4388_846f_85ee2f96e21c.slice/crio-ebc546672df1c22665ec9f4842e7729742d56cd50f29a9efef6c4eb76e5cc184 WatchSource:0}: Error finding container ebc546672df1c22665ec9f4842e7729742d56cd50f29a9efef6c4eb76e5cc184: Status 404 returned error can't find the container with id ebc546672df1c22665ec9f4842e7729742d56cd50f29a9efef6c4eb76e5cc184 Oct 30 02:50:08.491241 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:08.491201 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" event={"ID":"f6737d10-6c9f-4388-846f-85ee2f96e21c","Type":"ContainerStarted","Data":"75896c11027950cab79464d54849a9df692fa9b9b51c9e602b13f2eb27757a82"} Oct 30 02:50:08.491800 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:08.491256 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" event={"ID":"f6737d10-6c9f-4388-846f-85ee2f96e21c","Type":"ContainerStarted","Data":"ebc546672df1c22665ec9f4842e7729742d56cd50f29a9efef6c4eb76e5cc184"} Oct 30 02:50:08.532545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:08.532470 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" podStartSLOduration=1.532452116 podStartE2EDuration="1.532452116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:08.530839122 +0000 UTC m=+4887.033771342" watchObservedRunningTime="2026-10-30 02:50:08.532452116 +0000 UTC m=+4887.035384358" Oct 30 02:50:09.498349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:09.498309 5035 generic.go:356] "Generic (PLEG): container finished" podID="f6737d10-6c9f-4388-846f-85ee2f96e21c" containerID="75896c11027950cab79464d54849a9df692fa9b9b51c9e602b13f2eb27757a82" exitCode=0 Oct 30 02:50:09.498825 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:09.498379 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" event={"ID":"f6737d10-6c9f-4388-846f-85ee2f96e21c","Type":"ContainerDied","Data":"75896c11027950cab79464d54849a9df692fa9b9b51c9e602b13f2eb27757a82"} Oct 30 02:50:10.032016 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.031978 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk"] Oct 30 02:50:10.035087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.035055 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-913e8435-sgpkk"] Oct 30 02:50:10.648794 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.648763 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:10.763732 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.763629 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6737d10-6c9f-4388-846f-85ee2f96e21c-kube-api-access-8fxpn\" (UniqueName: \"kubernetes.io/projected/f6737d10-6c9f-4388-846f-85ee2f96e21c-kube-api-access-8fxpn\") pod \"f6737d10-6c9f-4388-846f-85ee2f96e21c\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " Oct 30 02:50:10.763732 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.763722 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-xks-ca\") pod \"f6737d10-6c9f-4388-846f-85ee2f96e21c\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " Oct 30 02:50:10.764011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.763885 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-openshift-ca\") pod \"f6737d10-6c9f-4388-846f-85ee2f96e21c\" (UID: \"f6737d10-6c9f-4388-846f-85ee2f96e21c\") " Oct 30 02:50:10.764262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.764228 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-openshift-ca" pod "f6737d10-6c9f-4388-846f-85ee2f96e21c" (UID: "f6737d10-6c9f-4388-846f-85ee2f96e21c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:50:10.765947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.765917 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-xks-ca" pod "f6737d10-6c9f-4388-846f-85ee2f96e21c" (UID: "f6737d10-6c9f-4388-846f-85ee2f96e21c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:50:10.766064 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.765985 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6737d10-6c9f-4388-846f-85ee2f96e21c-kube-api-access-8fxpn" pod "f6737d10-6c9f-4388-846f-85ee2f96e21c" (UID: "f6737d10-6c9f-4388-846f-85ee2f96e21c"). InnerVolumeSpecName "kube-api-access-8fxpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:50:10.865245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.865205 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:50:10.865245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.865238 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxpn\" (UniqueName: \"kubernetes.io/projected/f6737d10-6c9f-4388-846f-85ee2f96e21c-kube-api-access-8fxpn\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:50:10.865245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:10.865249 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6737d10-6c9f-4388-846f-85ee2f96e21c-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:50:11.510834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:11.510805 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" Oct 30 02:50:11.511029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:11.510836 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8" event={"ID":"f6737d10-6c9f-4388-846f-85ee2f96e21c","Type":"ContainerDied","Data":"ebc546672df1c22665ec9f4842e7729742d56cd50f29a9efef6c4eb76e5cc184"} Oct 30 02:50:11.511029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:11.510889 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebc546672df1c22665ec9f4842e7729742d56cd50f29a9efef6c4eb76e5cc184" Oct 30 02:50:11.641807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:11.641768 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea090139-df87-41c5-a7ca-6880c9ef8d8d" path="/var/lib/kubelet/pods/ea090139-df87-41c5-a7ca-6880c9ef8d8d/volumes" Oct 30 02:50:17.628702 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:50:17.628664 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:50:23.823757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.823668 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6"] Oct 30 02:50:23.824287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.824145 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f6737d10-6c9f-4388-846f-85ee2f96e21c" containerName="revoke-keys" Oct 30 02:50:23.824287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.824157 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6737d10-6c9f-4388-846f-85ee2f96e21c" containerName="revoke-keys" Oct 30 02:50:23.824287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.824229 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6737d10-6c9f-4388-846f-85ee2f96e21c" containerName="revoke-keys" Oct 30 02:50:23.828367 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.828345 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:23.859663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.858020 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:50:23.859663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.858021 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:50:23.859663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.858138 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:50:23.859663 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.858167 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:50:23.860898 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.860876 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6"] Oct 30 02:50:23.955110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.955060 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvsp6\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:23.955314 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.955131 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvsp6\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:23.955314 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:23.955240 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5zg\" (UniqueName: \"kubernetes.io/projected/bdcde04b-22d9-41c0-9e82-434834ddc708-kube-api-access-dw5zg\") pod \"maas-api-revoke-sub-0165d7d2-rvsp6\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:24.056887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:24.056842 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvsp6\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:24.057074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:24.056910 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvsp6\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:24.057074 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:24.056942 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5zg\" (UniqueName: \"kubernetes.io/projected/bdcde04b-22d9-41c0-9e82-434834ddc708-kube-api-access-dw5zg\") pod \"maas-api-revoke-sub-0165d7d2-rvsp6\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:24.057727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:24.057700 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvsp6\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:24.059456 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:24.059426 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvsp6\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:24.081010 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:24.080933 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5zg\" (UniqueName: \"kubernetes.io/projected/bdcde04b-22d9-41c0-9e82-434834ddc708-kube-api-access-dw5zg\") pod \"maas-api-revoke-sub-0165d7d2-rvsp6\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:24.142050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:24.142005 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:24.454868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:24.454835 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6"] Oct 30 02:50:24.456377 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:50:24.456345 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcde04b_22d9_41c0_9e82_434834ddc708.slice/crio-34053af79a04e71fdb5814e86a1e8bd66f75c71c5571424fdaef0f3366cc0a20 WatchSource:0}: Error finding container 34053af79a04e71fdb5814e86a1e8bd66f75c71c5571424fdaef0f3366cc0a20: Status 404 returned error can't find the container with id 34053af79a04e71fdb5814e86a1e8bd66f75c71c5571424fdaef0f3366cc0a20 Oct 30 02:50:24.660786 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:24.660750 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" event={"ID":"bdcde04b-22d9-41c0-9e82-434834ddc708","Type":"ContainerStarted","Data":"34053af79a04e71fdb5814e86a1e8bd66f75c71c5571424fdaef0f3366cc0a20"} Oct 30 02:50:25.667595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:25.667490 5035 generic.go:356] "Generic (PLEG): container finished" podID="bdcde04b-22d9-41c0-9e82-434834ddc708" containerID="81aac3409df5dc35a9d75444a94efff54082c034887c7c4d2cbb1cef8763f40f" exitCode=0 Oct 30 02:50:25.667595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:25.667566 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" event={"ID":"bdcde04b-22d9-41c0-9e82-434834ddc708","Type":"ContainerDied","Data":"81aac3409df5dc35a9d75444a94efff54082c034887c7c4d2cbb1cef8763f40f"} Oct 30 02:50:26.823572 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:26.823513 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:26.989242 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:26.989201 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-xks-ca\") pod \"bdcde04b-22d9-41c0-9e82-434834ddc708\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " Oct 30 02:50:26.989485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:26.989268 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-openshift-ca\") pod \"bdcde04b-22d9-41c0-9e82-434834ddc708\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " Oct 30 02:50:26.989485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:26.989366 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdcde04b-22d9-41c0-9e82-434834ddc708-kube-api-access-dw5zg\" (UniqueName: \"kubernetes.io/projected/bdcde04b-22d9-41c0-9e82-434834ddc708-kube-api-access-dw5zg\") pod \"bdcde04b-22d9-41c0-9e82-434834ddc708\" (UID: \"bdcde04b-22d9-41c0-9e82-434834ddc708\") " Oct 30 02:50:26.993006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:26.991659 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-openshift-ca" pod "bdcde04b-22d9-41c0-9e82-434834ddc708" (UID: "bdcde04b-22d9-41c0-9e82-434834ddc708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:50:26.997162 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:26.997112 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcde04b-22d9-41c0-9e82-434834ddc708-kube-api-access-dw5zg" pod "bdcde04b-22d9-41c0-9e82-434834ddc708" (UID: "bdcde04b-22d9-41c0-9e82-434834ddc708"). InnerVolumeSpecName "kube-api-access-dw5zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:50:27.000110 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:27.000074 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-xks-ca" pod "bdcde04b-22d9-41c0-9e82-434834ddc708" (UID: "bdcde04b-22d9-41c0-9e82-434834ddc708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:50:27.091378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:27.091331 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:50:27.091378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:27.091372 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcde04b-22d9-41c0-9e82-434834ddc708-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:50:27.091378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:27.091383 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw5zg\" (UniqueName: \"kubernetes.io/projected/bdcde04b-22d9-41c0-9e82-434834ddc708-kube-api-access-dw5zg\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:50:27.679289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:27.679256 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" Oct 30 02:50:27.679504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:27.679292 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6" event={"ID":"bdcde04b-22d9-41c0-9e82-434834ddc708","Type":"ContainerDied","Data":"34053af79a04e71fdb5814e86a1e8bd66f75c71c5571424fdaef0f3366cc0a20"} Oct 30 02:50:27.679504 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:27.679342 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34053af79a04e71fdb5814e86a1e8bd66f75c71c5571424fdaef0f3366cc0a20" Oct 30 02:50:34.754388 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.754347 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669"] Oct 30 02:50:34.754888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.754871 5035 cpu_manager.go:414] "containerMap: removing container" podUID="bdcde04b-22d9-41c0-9e82-434834ddc708" containerName="revoke-keys" Oct 30 02:50:34.754935 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.754889 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcde04b-22d9-41c0-9e82-434834ddc708" containerName="revoke-keys" Oct 30 02:50:34.754991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.754981 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcde04b-22d9-41c0-9e82-434834ddc708" containerName="revoke-keys" Oct 30 02:50:34.758382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.758363 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:34.771006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.770972 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3768308b-c017-4359-a598-a487135cf8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mg669\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:34.771179 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.771024 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3768308b-c017-4359-a598-a487135cf8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mg669\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:34.771179 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.771110 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtsb8\" (UniqueName: \"kubernetes.io/projected/3768308b-c017-4359-a598-a487135cf8fd-kube-api-access-wtsb8\") pod \"maas-api-revoke-sub-6b19338f-mg669\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:34.779558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.779507 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669"] Oct 30 02:50:34.787885 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.787851 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:50:34.788092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.787851 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:50:34.788092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.787902 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:50:34.788092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.787919 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:50:34.872213 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.872169 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3768308b-c017-4359-a598-a487135cf8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mg669\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:34.872440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.872226 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3768308b-c017-4359-a598-a487135cf8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mg669\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:34.872440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.872249 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsb8\" (UniqueName: \"kubernetes.io/projected/3768308b-c017-4359-a598-a487135cf8fd-kube-api-access-wtsb8\") pod \"maas-api-revoke-sub-6b19338f-mg669\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:34.873058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.873024 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3768308b-c017-4359-a598-a487135cf8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-mg669\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:34.874654 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.874631 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3768308b-c017-4359-a598-a487135cf8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-mg669\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:34.896379 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:34.896343 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsb8\" (UniqueName: \"kubernetes.io/projected/3768308b-c017-4359-a598-a487135cf8fd-kube-api-access-wtsb8\") pod \"maas-api-revoke-sub-6b19338f-mg669\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:35.072449 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:35.072341 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:35.384050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:35.384018 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669"] Oct 30 02:50:35.386247 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:50:35.386212 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3768308b_c017_4359_a598_a487135cf8fd.slice/crio-7ea8da656c46fa85af5e186296f73da8c7c46821700d6a1e37b4520a221074c3 WatchSource:0}: Error finding container 7ea8da656c46fa85af5e186296f73da8c7c46821700d6a1e37b4520a221074c3: Status 404 returned error can't find the container with id 7ea8da656c46fa85af5e186296f73da8c7c46821700d6a1e37b4520a221074c3 Oct 30 02:50:35.727522 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:35.727473 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" event={"ID":"3768308b-c017-4359-a598-a487135cf8fd","Type":"ContainerStarted","Data":"36d58554cbd6c9fde69da7467816c1b072c2e6a3a85dc98122390ab422b1c380"} Oct 30 02:50:35.727741 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:35.727557 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" event={"ID":"3768308b-c017-4359-a598-a487135cf8fd","Type":"ContainerStarted","Data":"7ea8da656c46fa85af5e186296f73da8c7c46821700d6a1e37b4520a221074c3"} Oct 30 02:50:35.771625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:35.771565 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" podStartSLOduration=1.7715210620000001 podStartE2EDuration="1.771521062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:35.768807547 +0000 UTC m=+4914.271739784" watchObservedRunningTime="2026-10-30 02:50:35.771521062 +0000 UTC m=+4914.274453329" Oct 30 02:50:36.736045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:36.736007 5035 generic.go:356] "Generic (PLEG): container finished" podID="3768308b-c017-4359-a598-a487135cf8fd" containerID="36d58554cbd6c9fde69da7467816c1b072c2e6a3a85dc98122390ab422b1c380" exitCode=0 Oct 30 02:50:36.736244 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:36.736073 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" event={"ID":"3768308b-c017-4359-a598-a487135cf8fd","Type":"ContainerDied","Data":"36d58554cbd6c9fde69da7467816c1b072c2e6a3a85dc98122390ab422b1c380"} Oct 30 02:50:37.896383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:37.896351 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:38.005397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.005289 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3768308b-c017-4359-a598-a487135cf8fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3768308b-c017-4359-a598-a487135cf8fd-maas-api-xks-ca\") pod \"3768308b-c017-4359-a598-a487135cf8fd\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " Oct 30 02:50:38.005614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.005395 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3768308b-c017-4359-a598-a487135cf8fd-kube-api-access-wtsb8\" (UniqueName: \"kubernetes.io/projected/3768308b-c017-4359-a598-a487135cf8fd-kube-api-access-wtsb8\") pod \"3768308b-c017-4359-a598-a487135cf8fd\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " Oct 30 02:50:38.005614 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.005431 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3768308b-c017-4359-a598-a487135cf8fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3768308b-c017-4359-a598-a487135cf8fd-maas-api-openshift-ca\") pod \"3768308b-c017-4359-a598-a487135cf8fd\" (UID: \"3768308b-c017-4359-a598-a487135cf8fd\") " Oct 30 02:50:38.005860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.005830 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3768308b-c017-4359-a598-a487135cf8fd-maas-api-openshift-ca" pod "3768308b-c017-4359-a598-a487135cf8fd" (UID: "3768308b-c017-4359-a598-a487135cf8fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:50:38.007475 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.007450 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3768308b-c017-4359-a598-a487135cf8fd-maas-api-xks-ca" pod "3768308b-c017-4359-a598-a487135cf8fd" (UID: "3768308b-c017-4359-a598-a487135cf8fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:50:38.007625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.007539 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3768308b-c017-4359-a598-a487135cf8fd-kube-api-access-wtsb8" pod "3768308b-c017-4359-a598-a487135cf8fd" (UID: "3768308b-c017-4359-a598-a487135cf8fd"). InnerVolumeSpecName "kube-api-access-wtsb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:50:38.106638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.106595 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3768308b-c017-4359-a598-a487135cf8fd-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:50:38.106638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.106633 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtsb8\" (UniqueName: \"kubernetes.io/projected/3768308b-c017-4359-a598-a487135cf8fd-kube-api-access-wtsb8\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:50:38.106638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.106643 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3768308b-c017-4359-a598-a487135cf8fd-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:50:38.750521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.750429 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" event={"ID":"3768308b-c017-4359-a598-a487135cf8fd","Type":"ContainerDied","Data":"7ea8da656c46fa85af5e186296f73da8c7c46821700d6a1e37b4520a221074c3"} Oct 30 02:50:38.750521 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.750507 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669" Oct 30 02:50:38.750978 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:38.750579 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ea8da656c46fa85af5e186296f73da8c7c46821700d6a1e37b4520a221074c3" Oct 30 02:50:47.720553 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:50:47.719603 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:50:49.598251 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.598202 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx"] Oct 30 02:50:49.598963 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.598939 5035 cpu_manager.go:414] "containerMap: removing container" podUID="3768308b-c017-4359-a598-a487135cf8fd" containerName="revoke-keys" Oct 30 02:50:49.599011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.598967 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3768308b-c017-4359-a598-a487135cf8fd" containerName="revoke-keys" Oct 30 02:50:49.599130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.599114 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="3768308b-c017-4359-a598-a487135cf8fd" containerName="revoke-keys" Oct 30 02:50:49.604842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.604757 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.622222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.622186 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx"] Oct 30 02:50:49.630031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.629993 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-8b3c5a4d-kserve-self-signed-certs\"" Oct 30 02:50:49.688630 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.688585 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-dshm\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.688826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.688642 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-home\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.688826 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.688744 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-model-cache\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.688904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.688857 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b9981f-fa71-4224-914b-54862c122cfe-tls-certs\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.688943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.688912 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4sh\" (UniqueName: \"kubernetes.io/projected/81b9981f-fa71-4224-914b-54862c122cfe-kube-api-access-5d4sh\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.688943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.688946 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-tmp-dir\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.790453 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.790404 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-tmp-dir\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.790662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.790461 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-dshm\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.790662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.790500 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-home\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.790662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.790560 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-model-cache\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.790662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.790652 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b9981f-fa71-4224-914b-54862c122cfe-tls-certs\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.790924 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.790719 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4sh\" (UniqueName: \"kubernetes.io/projected/81b9981f-fa71-4224-914b-54862c122cfe-kube-api-access-5d4sh\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.790996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.790950 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-tmp-dir\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.791094 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.791072 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-model-cache\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.791155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.791111 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-home\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.792865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.792831 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-dshm\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.793369 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.793350 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b9981f-fa71-4224-914b-54862c122cfe-tls-certs\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.814407 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.814368 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4sh\" (UniqueName: \"kubernetes.io/projected/81b9981f-fa71-4224-914b-54862c122cfe-kube-api-access-5d4sh\") pod \"e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:49.922604 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:49.922474 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:50.244718 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:50.244599 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx"] Oct 30 02:50:50.246856 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:50:50.246812 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b9981f_fa71_4224_914b_54862c122cfe.slice/crio-d12fb4a58dac32506cc35b8ae9d9be7e01410f1156bddd4c679027c1f5b2a07a WatchSource:0}: Error finding container d12fb4a58dac32506cc35b8ae9d9be7e01410f1156bddd4c679027c1f5b2a07a: Status 404 returned error can't find the container with id d12fb4a58dac32506cc35b8ae9d9be7e01410f1156bddd4c679027c1f5b2a07a Oct 30 02:50:50.848508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:50.848460 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" event={"ID":"81b9981f-fa71-4224-914b-54862c122cfe","Type":"ContainerStarted","Data":"c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8"} Oct 30 02:50:50.849003 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:50.848522 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" event={"ID":"81b9981f-fa71-4224-914b-54862c122cfe","Type":"ContainerStarted","Data":"d12fb4a58dac32506cc35b8ae9d9be7e01410f1156bddd4c679027c1f5b2a07a"} Oct 30 02:50:50.890835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:50.890779 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" podStartSLOduration=1.890762134 podStartE2EDuration="1.890762134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:50:50.889619239 +0000 UTC m=+4929.392551459" watchObservedRunningTime="2026-10-30 02:50:50.890762134 +0000 UTC m=+4929.393694348" Oct 30 02:50:50.922997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:50.922957 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:59.922796 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:59.922734 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:59.936215 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:59.936180 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:50:59.963138 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:50:59.963097 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:51:17.630480 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:51:17.630437 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:51:37.748767 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.748713 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb"] Oct 30 02:51:37.752962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.752934 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:37.769417 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.769382 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:51:37.781673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.781554 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:51:37.781673 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.781643 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:51:37.782157 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.782127 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:51:37.784721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.784687 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb"] Oct 30 02:51:37.810229 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.810178 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0549523-8xvpb\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:37.810438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.810265 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0549523-8xvpb\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:37.810438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.810368 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nlm\" (UniqueName: \"kubernetes.io/projected/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-kube-api-access-92nlm\") pod \"maas-api-revoke-sub-a0549523-8xvpb\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:37.911109 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.911064 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0549523-8xvpb\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:37.911338 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.911132 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0549523-8xvpb\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:37.911338 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.911157 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nlm\" (UniqueName: \"kubernetes.io/projected/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-kube-api-access-92nlm\") pod \"maas-api-revoke-sub-a0549523-8xvpb\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:37.911914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.911884 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0549523-8xvpb\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:37.913659 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.913634 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0549523-8xvpb\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:37.934806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:37.934775 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nlm\" (UniqueName: \"kubernetes.io/projected/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-kube-api-access-92nlm\") pod \"maas-api-revoke-sub-a0549523-8xvpb\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:38.067608 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:38.067478 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:38.377185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:38.377146 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb"] Oct 30 02:51:38.378851 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:51:38.378821 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff61e09_7a60_4cb4_b53a_8c7c9da66507.slice/crio-4867c21fbb0510d08f1dbf33c09518810eae58b5194aadd7eb85396f26b50ff9 WatchSource:0}: Error finding container 4867c21fbb0510d08f1dbf33c09518810eae58b5194aadd7eb85396f26b50ff9: Status 404 returned error can't find the container with id 4867c21fbb0510d08f1dbf33c09518810eae58b5194aadd7eb85396f26b50ff9 Oct 30 02:51:39.227687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:39.227652 5035 generic.go:356] "Generic (PLEG): container finished" podID="8ff61e09-7a60-4cb4-b53a-8c7c9da66507" containerID="ed7ef7dbf0a3ee361536c608c00134f38f0811b1cf9414f11ede5e9793e93803" exitCode=0 Oct 30 02:51:39.228106 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:39.227722 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" event={"ID":"8ff61e09-7a60-4cb4-b53a-8c7c9da66507","Type":"ContainerDied","Data":"ed7ef7dbf0a3ee361536c608c00134f38f0811b1cf9414f11ede5e9793e93803"} Oct 30 02:51:39.228106 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:39.227763 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" event={"ID":"8ff61e09-7a60-4cb4-b53a-8c7c9da66507","Type":"ContainerStarted","Data":"4867c21fbb0510d08f1dbf33c09518810eae58b5194aadd7eb85396f26b50ff9"} Oct 30 02:51:40.392878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.392844 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:40.438169 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.438133 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-kube-api-access-92nlm\" (UniqueName: \"kubernetes.io/projected/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-kube-api-access-92nlm\") pod \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " Oct 30 02:51:40.438383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.438281 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-openshift-ca\") pod \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " Oct 30 02:51:40.438383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.438323 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-xks-ca\") pod \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\" (UID: \"8ff61e09-7a60-4cb4-b53a-8c7c9da66507\") " Oct 30 02:51:40.438733 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.438700 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-openshift-ca" pod "8ff61e09-7a60-4cb4-b53a-8c7c9da66507" (UID: "8ff61e09-7a60-4cb4-b53a-8c7c9da66507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:51:40.440894 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.440868 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-kube-api-access-92nlm" pod "8ff61e09-7a60-4cb4-b53a-8c7c9da66507" (UID: "8ff61e09-7a60-4cb4-b53a-8c7c9da66507"). InnerVolumeSpecName "kube-api-access-92nlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:51:40.443660 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.443630 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-xks-ca" pod "8ff61e09-7a60-4cb4-b53a-8c7c9da66507" (UID: "8ff61e09-7a60-4cb4-b53a-8c7c9da66507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:51:40.539918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.539810 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:51:40.539918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.539855 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:51:40.539918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:40.539869 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nlm\" (UniqueName: \"kubernetes.io/projected/8ff61e09-7a60-4cb4-b53a-8c7c9da66507-kube-api-access-92nlm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:51:41.241632 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:41.241591 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" Oct 30 02:51:41.241852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:41.241590 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb" event={"ID":"8ff61e09-7a60-4cb4-b53a-8c7c9da66507","Type":"ContainerDied","Data":"4867c21fbb0510d08f1dbf33c09518810eae58b5194aadd7eb85396f26b50ff9"} Oct 30 02:51:41.241852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:41.241763 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4867c21fbb0510d08f1dbf33c09518810eae58b5194aadd7eb85396f26b50ff9" Oct 30 02:51:44.953849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:44.953810 5035 scope.go:122] "RemoveContainer" containerID="bf326ac648e7b3ddd1b7a9e885b10a4f0cb2dd3d297c0bcc7983319bd25c5dc2" Oct 30 02:51:47.629409 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:51:47.629365 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:51:49.611032 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:49.610986 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx"] Oct 30 02:51:52.463099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.463059 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj"] Oct 30 02:51:52.463568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.463553 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8ff61e09-7a60-4cb4-b53a-8c7c9da66507" containerName="revoke-keys" Oct 30 02:51:52.463628 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.463570 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff61e09-7a60-4cb4-b53a-8c7c9da66507" containerName="revoke-keys" Oct 30 02:51:52.463667 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.463638 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff61e09-7a60-4cb4-b53a-8c7c9da66507" containerName="revoke-keys" Oct 30 02:51:52.468346 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.468319 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.490251 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.490206 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj"] Oct 30 02:51:52.498168 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.498124 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:51:52.498372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.498290 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:51:52.498372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.498347 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:51:52.498489 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.498425 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:51:52.577881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.577829 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea923b9-7v5sj\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.578099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.577903 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea923b9-7v5sj\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.578099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.578038 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpk7\" (UniqueName: \"kubernetes.io/projected/b139cf64-e57c-4a19-a31c-8fc68e6155f7-kube-api-access-qvpk7\") pod \"maas-api-revoke-sub-aea923b9-7v5sj\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.679130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.679081 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea923b9-7v5sj\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.679341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.679146 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea923b9-7v5sj\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.679341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.679216 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpk7\" (UniqueName: \"kubernetes.io/projected/b139cf64-e57c-4a19-a31c-8fc68e6155f7-kube-api-access-qvpk7\") pod \"maas-api-revoke-sub-aea923b9-7v5sj\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.679890 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.679867 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea923b9-7v5sj\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.681563 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.681543 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea923b9-7v5sj\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.703061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.703022 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpk7\" (UniqueName: \"kubernetes.io/projected/b139cf64-e57c-4a19-a31c-8fc68e6155f7-kube-api-access-qvpk7\") pod \"maas-api-revoke-sub-aea923b9-7v5sj\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:52.782899 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:52.782801 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:53.096852 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:53.096806 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj"] Oct 30 02:51:53.099059 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:51:53.099016 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb139cf64_e57c_4a19_a31c_8fc68e6155f7.slice/crio-cfde07d728a66d7ae12724d407e45188f58a0e8ba1b3425a54349a9bf3c045f7 WatchSource:0}: Error finding container cfde07d728a66d7ae12724d407e45188f58a0e8ba1b3425a54349a9bf3c045f7: Status 404 returned error can't find the container with id cfde07d728a66d7ae12724d407e45188f58a0e8ba1b3425a54349a9bf3c045f7 Oct 30 02:51:53.315102 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:53.315051 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" event={"ID":"b139cf64-e57c-4a19-a31c-8fc68e6155f7","Type":"ContainerStarted","Data":"cfde07d728a66d7ae12724d407e45188f58a0e8ba1b3425a54349a9bf3c045f7"} Oct 30 02:51:54.413035 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:54.409110 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" event={"ID":"b139cf64-e57c-4a19-a31c-8fc68e6155f7","Type":"ContainerStarted","Data":"7a67cd7e45acabbef7fb36524d2c2d00ef499c044e2ce5e168bf1a8db48dcf4d"} Oct 30 02:51:55.450568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:55.450072 5035 generic.go:356] "Generic (PLEG): container finished" podID="b139cf64-e57c-4a19-a31c-8fc68e6155f7" containerID="7a67cd7e45acabbef7fb36524d2c2d00ef499c044e2ce5e168bf1a8db48dcf4d" exitCode=0 Oct 30 02:51:55.450568 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:55.450295 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" event={"ID":"b139cf64-e57c-4a19-a31c-8fc68e6155f7","Type":"ContainerDied","Data":"7a67cd7e45acabbef7fb36524d2c2d00ef499c044e2ce5e168bf1a8db48dcf4d"} Oct 30 02:51:56.604236 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.604203 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:56.670086 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.670048 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b139cf64-e57c-4a19-a31c-8fc68e6155f7-kube-api-access-qvpk7\" (UniqueName: \"kubernetes.io/projected/b139cf64-e57c-4a19-a31c-8fc68e6155f7-kube-api-access-qvpk7\") pod \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " Oct 30 02:51:56.670274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.670101 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-openshift-ca\") pod \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " Oct 30 02:51:56.670274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.670144 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-xks-ca\") pod \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\" (UID: \"b139cf64-e57c-4a19-a31c-8fc68e6155f7\") " Oct 30 02:51:56.670585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.670558 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-openshift-ca" pod "b139cf64-e57c-4a19-a31c-8fc68e6155f7" (UID: "b139cf64-e57c-4a19-a31c-8fc68e6155f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:51:56.672352 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.672327 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-xks-ca" pod "b139cf64-e57c-4a19-a31c-8fc68e6155f7" (UID: "b139cf64-e57c-4a19-a31c-8fc68e6155f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:51:56.672486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.672457 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b139cf64-e57c-4a19-a31c-8fc68e6155f7-kube-api-access-qvpk7" pod "b139cf64-e57c-4a19-a31c-8fc68e6155f7" (UID: "b139cf64-e57c-4a19-a31c-8fc68e6155f7"). InnerVolumeSpecName "kube-api-access-qvpk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:51:56.771027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.770937 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvpk7\" (UniqueName: \"kubernetes.io/projected/b139cf64-e57c-4a19-a31c-8fc68e6155f7-kube-api-access-qvpk7\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:51:56.771027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.770970 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:51:56.771027 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:56.770981 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b139cf64-e57c-4a19-a31c-8fc68e6155f7-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:51:57.464005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:57.463970 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" Oct 30 02:51:57.464222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:57.463976 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj" event={"ID":"b139cf64-e57c-4a19-a31c-8fc68e6155f7","Type":"ContainerDied","Data":"cfde07d728a66d7ae12724d407e45188f58a0e8ba1b3425a54349a9bf3c045f7"} Oct 30 02:51:57.464222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:51:57.464105 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfde07d728a66d7ae12724d407e45188f58a0e8ba1b3425a54349a9bf3c045f7" Oct 30 02:52:04.632912 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.632859 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" podUID="81b9981f-fa71-4224-914b-54862c122cfe" containerName="main" containerID="cri-o://c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8" gracePeriod=45 Oct 30 02:52:04.794484 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.794439 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c"] Oct 30 02:52:04.794952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.794936 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b139cf64-e57c-4a19-a31c-8fc68e6155f7" containerName="revoke-keys" Oct 30 02:52:04.795013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.794954 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b139cf64-e57c-4a19-a31c-8fc68e6155f7" containerName="revoke-keys" Oct 30 02:52:04.795091 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.795048 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b139cf64-e57c-4a19-a31c-8fc68e6155f7" containerName="revoke-keys" Oct 30 02:52:04.798638 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.798608 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.812914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.812878 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d31d8-4a96-490e-98de-5b1926986894-istio-podinfo\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.812914 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.812922 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxgr\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-kube-api-access-4vxgr\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.813165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.813029 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-envoy\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.813165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.813089 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-credential-socket\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.813291 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.813189 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-certs\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.813356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.813309 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-data\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.813421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.813360 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-socket\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.813421 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.813393 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-istio-token\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.813623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.813606 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d31d8-4a96-490e-98de-5b1926986894-istiod-ca-cert\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.837353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.837295 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-88fa81-openshift-default-dockercfg-kqjsq\"" Oct 30 02:52:04.842056 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.842017 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c"] Oct 30 02:52:04.914633 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.914589 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-credential-socket\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.914803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.914669 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-certs\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.914803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.914731 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-data\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.914803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.914769 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-socket\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.914803 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.914800 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-istio-token\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.915012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.914841 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d31d8-4a96-490e-98de-5b1926986894-istiod-ca-cert\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.915012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.914916 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d31d8-4a96-490e-98de-5b1926986894-istio-podinfo\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.915135 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.915049 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-credential-socket\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.915135 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.915116 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-certs\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.915256 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.915146 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-data\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.915317 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.915268 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-socket\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.915486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.914948 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxgr\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-kube-api-access-4vxgr\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.915625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.915600 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-envoy\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.915707 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.915685 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d31d8-4a96-490e-98de-5b1926986894-istiod-ca-cert\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.917892 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.917864 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d31d8-4a96-490e-98de-5b1926986894-istio-podinfo\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.918053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.918026 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-envoy\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.945352 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.945302 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:52:04.989923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.989888 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-istio-token\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:04.990241 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:04.990218 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxgr\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-kube-api-access-4vxgr\") pod \"e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:05.017053 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017013 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-dshm\") pod \"81b9981f-fa71-4224-914b-54862c122cfe\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " Oct 30 02:52:05.017239 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017092 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-model-cache\") pod \"81b9981f-fa71-4224-914b-54862c122cfe\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " Oct 30 02:52:05.017239 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017143 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-tmp-dir\") pod \"81b9981f-fa71-4224-914b-54862c122cfe\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " Oct 30 02:52:05.017239 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017228 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b9981f-fa71-4224-914b-54862c122cfe-kube-api-access-5d4sh\" (UniqueName: \"kubernetes.io/projected/81b9981f-fa71-4224-914b-54862c122cfe-kube-api-access-5d4sh\") pod \"81b9981f-fa71-4224-914b-54862c122cfe\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " Oct 30 02:52:05.017419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017264 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-home\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-home\") pod \"81b9981f-fa71-4224-914b-54862c122cfe\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " Oct 30 02:52:05.017419 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017354 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81b9981f-fa71-4224-914b-54862c122cfe-tls-certs\" (UniqueName: \"kubernetes.io/secret/81b9981f-fa71-4224-914b-54862c122cfe-tls-certs\") pod \"81b9981f-fa71-4224-914b-54862c122cfe\" (UID: \"81b9981f-fa71-4224-914b-54862c122cfe\") " Oct 30 02:52:05.017559 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017350 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-model-cache" pod "81b9981f-fa71-4224-914b-54862c122cfe" (UID: "81b9981f-fa71-4224-914b-54862c122cfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:52:05.017662 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017639 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-home" pod "81b9981f-fa71-4224-914b-54862c122cfe" (UID: "81b9981f-fa71-4224-914b-54862c122cfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:52:05.017883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017745 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:52:05.017883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.017793 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-tmp-dir" pod "81b9981f-fa71-4224-914b-54862c122cfe" (UID: "81b9981f-fa71-4224-914b-54862c122cfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:52:05.019828 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.019802 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-dshm" pod "81b9981f-fa71-4224-914b-54862c122cfe" (UID: "81b9981f-fa71-4224-914b-54862c122cfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:52:05.020012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.019993 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b9981f-fa71-4224-914b-54862c122cfe-kube-api-access-5d4sh" pod "81b9981f-fa71-4224-914b-54862c122cfe" (UID: "81b9981f-fa71-4224-914b-54862c122cfe"). InnerVolumeSpecName "kube-api-access-5d4sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:52:05.020687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.020659 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b9981f-fa71-4224-914b-54862c122cfe-tls-certs" pod "81b9981f-fa71-4224-914b-54862c122cfe" (UID: "81b9981f-fa71-4224-914b-54862c122cfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:52:05.114773 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.114728 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:05.119257 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.119232 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:52:05.119323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.119261 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:52:05.119323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.119271 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4sh\" (UniqueName: \"kubernetes.io/projected/81b9981f-fa71-4224-914b-54862c122cfe-kube-api-access-5d4sh\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:52:05.119323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.119280 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b9981f-fa71-4224-914b-54862c122cfe-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:52:05.119323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.119290 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b9981f-fa71-4224-914b-54862c122cfe-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:52:05.443754 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:52:05.443716 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1d31d8_4a96_490e_98de_5b1926986894.slice/crio-99c72df60aaa61d7d42a9c25c70b8ac9e2ef6d274a5823c3245cfeb902e5cb5b WatchSource:0}: Error finding container 99c72df60aaa61d7d42a9c25c70b8ac9e2ef6d274a5823c3245cfeb902e5cb5b: Status 404 returned error can't find the container with id 99c72df60aaa61d7d42a9c25c70b8ac9e2ef6d274a5823c3245cfeb902e5cb5b Oct 30 02:52:05.445045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.445017 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c"] Oct 30 02:52:05.445956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.445921 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:52:05.446045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.445990 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:52:05.446045 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.446019 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:52:05.544316 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.544278 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" event={"ID":"8e1d31d8-4a96-490e-98de-5b1926986894","Type":"ContainerStarted","Data":"9ddabd16d2c0e721c661987473d7bc6982b49657cc16159dd0b955c6ceef6509"} Oct 30 02:52:05.544561 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.544348 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" event={"ID":"8e1d31d8-4a96-490e-98de-5b1926986894","Type":"ContainerStarted","Data":"99c72df60aaa61d7d42a9c25c70b8ac9e2ef6d274a5823c3245cfeb902e5cb5b"} Oct 30 02:52:05.546087 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.546050 5035 generic.go:356] "Generic (PLEG): container finished" podID="81b9981f-fa71-4224-914b-54862c122cfe" containerID="c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8" exitCode=0 Oct 30 02:52:05.546286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.546134 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" Oct 30 02:52:05.546286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.546145 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" event={"ID":"81b9981f-fa71-4224-914b-54862c122cfe","Type":"ContainerDied","Data":"c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8"} Oct 30 02:52:05.546286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.546187 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx" event={"ID":"81b9981f-fa71-4224-914b-54862c122cfe","Type":"ContainerDied","Data":"d12fb4a58dac32506cc35b8ae9d9be7e01410f1156bddd4c679027c1f5b2a07a"} Oct 30 02:52:05.546286 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.546217 5035 scope.go:122] "RemoveContainer" containerID="c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8" Oct 30 02:52:05.558326 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.558296 5035 scope.go:122] "RemoveContainer" containerID="c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8" Oct 30 02:52:05.558731 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:52:05.558705 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8\": container with ID starting with c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8 not found: ID does not exist" containerID="c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8" Oct 30 02:52:05.558815 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.558742 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8"} err="failed to get container status \"c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8\": rpc error: code = NotFound desc = could not find container \"c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8\": container with ID starting with c4e6038dee341e620641147ab1af88426990ef1d9523c7544662187696461fb8 not found: ID does not exist" Oct 30 02:52:05.591920 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.591846 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" podStartSLOduration=1.5918272199999999 podStartE2EDuration="1.59182722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:05.590590591 +0000 UTC m=+5004.093522815" watchObservedRunningTime="2026-10-30 02:52:05.59182722 +0000 UTC m=+5004.094759443" Oct 30 02:52:05.628873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.628835 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx"] Oct 30 02:52:05.632655 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.632620 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-8b3c5a4d-kserve-fd74466b5-wjrgx"] Oct 30 02:52:05.642872 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:05.642837 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b9981f-fa71-4224-914b-54862c122cfe" path="/var/lib/kubelet/pods/81b9981f-fa71-4224-914b-54862c122cfe/volumes" Oct 30 02:52:06.115980 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:06.115941 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:06.121192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:06.121158 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:06.553930 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:06.553896 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:06.554963 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:06.554944 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:09.465809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.465771 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744"] Oct 30 02:52:09.466323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.466221 5035 cpu_manager.go:414] "containerMap: removing container" podUID="81b9981f-fa71-4224-914b-54862c122cfe" containerName="main" Oct 30 02:52:09.466323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.466233 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b9981f-fa71-4224-914b-54862c122cfe" containerName="main" Oct 30 02:52:09.466408 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.466329 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b9981f-fa71-4224-914b-54862c122cfe" containerName="main" Oct 30 02:52:09.471386 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.471363 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.502645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.502425 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:52:09.502645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.502514 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-88fa81-serving-cert\"" Oct 30 02:52:09.502645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.502578 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:52:09.503161 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.503129 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 02:52:09.503765 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.503732 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xzt6k\"" Oct 30 02:52:09.506966 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.506939 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744"] Oct 30 02:52:09.567731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.567692 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a098155d-bc55-482b-8556-9c76db1fc5a3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.567731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.567736 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.567997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.567883 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.567997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.567973 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtkvl\" (UniqueName: \"kubernetes.io/projected/a098155d-bc55-482b-8556-9c76db1fc5a3-kube-api-access-rtkvl\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.669367 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.669322 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.669622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.669404 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtkvl\" (UniqueName: \"kubernetes.io/projected/a098155d-bc55-482b-8556-9c76db1fc5a3-kube-api-access-rtkvl\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.669622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.669563 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a098155d-bc55-482b-8556-9c76db1fc5a3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.669622 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.669612 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.670585 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.670550 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a098155d-bc55-482b-8556-9c76db1fc5a3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.672313 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.672280 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.672413 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.672350 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.693210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.693175 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtkvl\" (UniqueName: \"kubernetes.io/projected/a098155d-bc55-482b-8556-9c76db1fc5a3-kube-api-access-rtkvl\") pod \"maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.784804 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.784707 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:09.904465 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.904413 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj"] Oct 30 02:52:09.910364 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.910332 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:09.930054 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.930015 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-88fa81\"" Oct 30 02:52:09.930696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.930674 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-1b48ea6b-dockercfg-qfkbd\"" Oct 30 02:52:09.935725 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.935686 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj"] Oct 30 02:52:09.963805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.963764 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8"] Oct 30 02:52:09.969735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.969694 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:09.972586 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.972549 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ac2b09-8aae-4f5a-a842-b74e70d50e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj\" (UID: \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:09.972783 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:09.972630 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9skr\" (UniqueName: \"kubernetes.io/projected/43ac2b09-8aae-4f5a-a842-b74e70d50e39-kube-api-access-b9skr\") pod \"payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj\" (UID: \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:10.010026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.002033 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8"] Oct 30 02:52:10.074143 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.074030 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ac2b09-8aae-4f5a-a842-b74e70d50e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj\" (UID: \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:10.074143 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.074113 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9skr\" (UniqueName: \"kubernetes.io/projected/43ac2b09-8aae-4f5a-a842-b74e70d50e39-kube-api-access-b9skr\") pod \"payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj\" (UID: \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:10.074357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.074176 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b01c4-c63b-432f-b5b3-887b02a209d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8\" (UID: \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:10.074357 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.074262 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgl6l\" (UniqueName: \"kubernetes.io/projected/0b1b01c4-c63b-432f-b5b3-887b02a209d4-kube-api-access-vgl6l\") pod \"payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8\" (UID: \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:10.074829 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.074807 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ac2b09-8aae-4f5a-a842-b74e70d50e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj\" (UID: \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:10.100982 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.100948 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9skr\" (UniqueName: \"kubernetes.io/projected/43ac2b09-8aae-4f5a-a842-b74e70d50e39-kube-api-access-b9skr\") pod \"payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj\" (UID: \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:10.121843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.121811 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744"] Oct 30 02:52:10.123203 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:52:10.123172 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda098155d_bc55_482b_8556_9c76db1fc5a3.slice/crio-59f4b76014cf93acf10137ae223865a7d7f39e5f08057cd99afde50a1e9dca7e WatchSource:0}: Error finding container 59f4b76014cf93acf10137ae223865a7d7f39e5f08057cd99afde50a1e9dca7e: Status 404 returned error can't find the container with id 59f4b76014cf93acf10137ae223865a7d7f39e5f08057cd99afde50a1e9dca7e Oct 30 02:52:10.124809 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.124787 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:52:10.175200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.175147 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b01c4-c63b-432f-b5b3-887b02a209d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8\" (UID: \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:10.175200 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.175203 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgl6l\" (UniqueName: \"kubernetes.io/projected/0b1b01c4-c63b-432f-b5b3-887b02a209d4-kube-api-access-vgl6l\") pod \"payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8\" (UID: \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:10.176042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.176017 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b01c4-c63b-432f-b5b3-887b02a209d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8\" (UID: \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:10.207681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.207639 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgl6l\" (UniqueName: \"kubernetes.io/projected/0b1b01c4-c63b-432f-b5b3-887b02a209d4-kube-api-access-vgl6l\") pod \"payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8\" (UID: \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:10.231492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.231454 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:10.299656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.299618 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:10.545761 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.545729 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj"] Oct 30 02:52:10.547240 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:52:10.547201 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ac2b09_8aae_4f5a_a842_b74e70d50e39.slice/crio-ca4c83af652e52066da4bb11fcbb8a9848196c6885de81ffc4a700638c9135df WatchSource:0}: Error finding container ca4c83af652e52066da4bb11fcbb8a9848196c6885de81ffc4a700638c9135df: Status 404 returned error can't find the container with id ca4c83af652e52066da4bb11fcbb8a9848196c6885de81ffc4a700638c9135df Oct 30 02:52:10.580620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.580567 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" event={"ID":"a098155d-bc55-482b-8556-9c76db1fc5a3","Type":"ContainerStarted","Data":"59f4b76014cf93acf10137ae223865a7d7f39e5f08057cd99afde50a1e9dca7e"} Oct 30 02:52:10.582022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.581995 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" event={"ID":"43ac2b09-8aae-4f5a-a842-b74e70d50e39","Type":"ContainerStarted","Data":"ca4c83af652e52066da4bb11fcbb8a9848196c6885de81ffc4a700638c9135df"} Oct 30 02:52:10.619630 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:10.619576 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8"] Oct 30 02:52:10.623398 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:52:10.623364 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1b01c4_c63b_432f_b5b3_887b02a209d4.slice/crio-b0248ce507629c557435d8f8b545635f0dec86347bb6e675fa7a3f35e9b51260 WatchSource:0}: Error finding container b0248ce507629c557435d8f8b545635f0dec86347bb6e675fa7a3f35e9b51260: Status 404 returned error can't find the container with id b0248ce507629c557435d8f8b545635f0dec86347bb6e675fa7a3f35e9b51260 Oct 30 02:52:11.590686 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.590639 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" event={"ID":"43ac2b09-8aae-4f5a-a842-b74e70d50e39","Type":"ContainerStarted","Data":"ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d"} Oct 30 02:52:11.591224 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.590736 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:11.592539 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.592500 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" event={"ID":"0b1b01c4-c63b-432f-b5b3-887b02a209d4","Type":"ContainerStarted","Data":"2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281"} Oct 30 02:52:11.592680 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.592584 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:11.592680 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.592610 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" event={"ID":"0b1b01c4-c63b-432f-b5b3-887b02a209d4","Type":"ContainerStarted","Data":"b0248ce507629c557435d8f8b545635f0dec86347bb6e675fa7a3f35e9b51260"} Oct 30 02:52:11.594060 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.594035 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" event={"ID":"a098155d-bc55-482b-8556-9c76db1fc5a3","Type":"ContainerStarted","Data":"6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c"} Oct 30 02:52:11.594181 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.594159 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:11.633718 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.633656 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" podStartSLOduration=2.633639461 podStartE2EDuration="2.633639461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:11.631674732 +0000 UTC m=+5010.134606946" watchObservedRunningTime="2026-10-30 02:52:11.633639461 +0000 UTC m=+5010.136571681" Oct 30 02:52:11.674460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.674395 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" podStartSLOduration=2.674380758 podStartE2EDuration="2.674380758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:11.671486502 +0000 UTC m=+5010.174418722" watchObservedRunningTime="2026-10-30 02:52:11.674380758 +0000 UTC m=+5010.177312978" Oct 30 02:52:11.717617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:11.717558 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" podStartSLOduration=2.26604818 podStartE2EDuration="2.717543436s" totalImagesPullingTime="451.495256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:11.715170943 +0000 UTC m=+5010.218103177" watchObservedRunningTime="2026-10-30 02:52:11.717543436 +0000 UTC m=+5010.220475666" Oct 30 02:52:12.044105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:12.044065 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c"] Oct 30 02:52:12.047642 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:12.047606 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-765f4b3e-6zr2c"] Oct 30 02:52:13.641620 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:13.641582 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb" path="/var/lib/kubelet/pods/cd77c46c-19ee-4941-b3f7-4c7c3c2a47bb/volumes" Oct 30 02:52:17.607965 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:17.607936 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:52:17.629832 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:52:17.629783 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:52:19.094182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:19.094142 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp"] Oct 30 02:52:19.105972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:19.105931 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f09ebe-t79sp"] Oct 30 02:52:19.646619 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:19.646578 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b168647-e1e7-45e3-8057-d38459e03d61" path="/var/lib/kubelet/pods/0b168647-e1e7-45e3-8057-d38459e03d61/volumes" Oct 30 02:52:22.595731 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:22.595681 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:52:22.596299 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:22.595775 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:52:22.603985 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:22.603944 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:52:22.605220 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:22.605196 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:52:23.683588 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:23.683461 5035 generic.go:356] "Generic (PLEG): container finished" podID="8e1d31d8-4a96-490e-98de-5b1926986894" containerID="9ddabd16d2c0e721c661987473d7bc6982b49657cc16159dd0b955c6ceef6509" exitCode=0 Oct 30 02:52:23.683588 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:23.683551 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" event={"ID":"8e1d31d8-4a96-490e-98de-5b1926986894","Type":"ContainerDied","Data":"9ddabd16d2c0e721c661987473d7bc6982b49657cc16159dd0b955c6ceef6509"} Oct 30 02:52:23.684222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:23.684066 5035 scope.go:122] "RemoveContainer" containerID="9ddabd16d2c0e721c661987473d7bc6982b49657cc16159dd0b955c6ceef6509" Oct 30 02:52:23.685385 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:23.685358 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:52:23.685485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:23.685422 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:52:23.685485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:23.685457 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:52:24.702489 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:24.702445 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" event={"ID":"8e1d31d8-4a96-490e-98de-5b1926986894","Type":"ContainerStarted","Data":"1769d0b8ba261e9459c6779f31dc7796d27f61ff5f7d5d17bcbc47c9c71cdc7f"} Oct 30 02:52:25.116023 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:25.115930 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:25.116194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:25.116156 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 02:52:25.116243 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:25.116201 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 02:52:26.037206 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:26.037171 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8"] Oct 30 02:52:26.049188 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:26.049155 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69cefe28-j9lk8"] Oct 30 02:52:26.115592 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:26.115510 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 02:52:26.115812 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:26.115643 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 02:52:27.116022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.115980 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 02:52:27.116450 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.116044 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 02:52:27.143481 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.143418 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg"] Oct 30 02:52:27.147403 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.147379 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.192551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.166720 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-88fa81\"/\"openshift-service-ca.crt\"" Oct 30 02:52:27.192551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.170823 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-88fa81\"/\"kube-root-ca.crt\"" Oct 30 02:52:27.192551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.182809 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-88fa81\"/\"simulator-single-main-88fa81-kserve-self-signed-certs\"" Oct 30 02:52:27.192551 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.183002 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-88fa81\"/\"default-dockercfg-7kk2k\"" Oct 30 02:52:27.199618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.199274 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg"] Oct 30 02:52:27.313092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.313045 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-model-cache\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.313322 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.313107 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-dshm\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.313322 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.313307 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6fcn\" (UniqueName: \"kubernetes.io/projected/0b52547c-2763-4ed8-9960-da30f0b0a9d2-kube-api-access-v6fcn\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.313438 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.313349 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tmp-dir\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.313507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.313487 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-home\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.313600 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.313521 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tls-certs\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.414606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.414473 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-home\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.414606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.414516 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tls-certs\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.414606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.414573 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-model-cache\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.414606 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.414597 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-dshm\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.414921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.414719 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6fcn\" (UniqueName: \"kubernetes.io/projected/0b52547c-2763-4ed8-9960-da30f0b0a9d2-kube-api-access-v6fcn\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.414921 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.414754 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tmp-dir\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.415007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.414938 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-home\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.415051 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.415022 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-model-cache\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.415115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.415099 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tmp-dir\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.417104 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.417076 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-dshm\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.417378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.417361 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tls-certs\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.438952 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.438919 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6fcn\" (UniqueName: \"kubernetes.io/projected/0b52547c-2763-4ed8-9960-da30f0b0a9d2-kube-api-access-v6fcn\") pod \"simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.460895 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.460847 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:27.643639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.643598 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f439cc0-719a-4a9d-9881-24605b04af37" path="/var/lib/kubelet/pods/1f439cc0-719a-4a9d-9881-24605b04af37/volumes" Oct 30 02:52:27.812793 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:27.812428 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg"] Oct 30 02:52:27.814772 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:52:27.814736 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b52547c_2763_4ed8_9960_da30f0b0a9d2.slice/crio-939c10644cdb4e814eb947a0a550adda9759ecd9dda6cd27a1ba2c1a1797fec9 WatchSource:0}: Error finding container 939c10644cdb4e814eb947a0a550adda9759ecd9dda6cd27a1ba2c1a1797fec9: Status 404 returned error can't find the container with id 939c10644cdb4e814eb947a0a550adda9759ecd9dda6cd27a1ba2c1a1797fec9 Oct 30 02:52:28.115721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:28.115632 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 02:52:28.115721 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:28.115683 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 02:52:28.726735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:28.726695 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" event={"ID":"0b52547c-2763-4ed8-9960-da30f0b0a9d2","Type":"ContainerStarted","Data":"685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93"} Oct 30 02:52:28.727179 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:28.726746 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" event={"ID":"0b52547c-2763-4ed8-9960-da30f0b0a9d2","Type":"ContainerStarted","Data":"939c10644cdb4e814eb947a0a550adda9759ecd9dda6cd27a1ba2c1a1797fec9"} Oct 30 02:52:28.791617 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:28.782251 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" podStartSLOduration=1.782229987 podStartE2EDuration="1.782229987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:52:28.77898194 +0000 UTC m=+5027.281914188" watchObservedRunningTime="2026-10-30 02:52:28.782229987 +0000 UTC m=+5027.285162207" Oct 30 02:52:29.115222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:29.115126 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 30 02:52:29.115222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:29.115188 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 30 02:52:29.461870 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:29.461812 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:30.049292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:30.049237 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr"] Oct 30 02:52:30.060795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:30.060749 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d9527a-xp2qr"] Oct 30 02:52:30.501488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:30.501442 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:30.502033 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:30.502011 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:30.502833 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:30.502807 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:52:31.643349 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:31.643314 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027b1899-8034-49b9-9ae9-6a29f15a03f8" path="/var/lib/kubelet/pods/027b1899-8034-49b9-9ae9-6a29f15a03f8/volumes" Oct 30 02:52:37.461201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:37.461151 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:37.474088 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:37.474053 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:37.795682 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:37.795593 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:52:38.033785 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:38.033749 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt"] Oct 30 02:52:38.036338 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:38.036306 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849af60a-jj8zt"] Oct 30 02:52:39.643113 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:52:39.643073 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d626614-4ffd-4020-a2c6-987a96bd21dd" path="/var/lib/kubelet/pods/3d626614-4ffd-4020-a2c6-987a96bd21dd/volumes" Oct 30 02:52:47.629486 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:52:47.629445 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:53:13.520989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:13.520943 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg"] Oct 30 02:53:14.510727 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:14.510680 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8"] Oct 30 02:53:14.542343 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:14.542297 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj"] Oct 30 02:53:16.781671 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:16.781626 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744"] Oct 30 02:53:16.782182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:16.781903 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" podUID="a098155d-bc55-482b-8556-9c76db1fc5a3" containerName="maas-api" containerID="cri-o://6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c" gracePeriod=30 Oct 30 02:53:17.281291 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.281261 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:53:17.399370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.399262 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-maas-api-tls\") pod \"a098155d-bc55-482b-8556-9c76db1fc5a3\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " Oct 30 02:53:17.399370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.399312 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-metrics-certs\") pod \"a098155d-bc55-482b-8556-9c76db1fc5a3\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " Oct 30 02:53:17.399370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.399341 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a098155d-bc55-482b-8556-9c76db1fc5a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a098155d-bc55-482b-8556-9c76db1fc5a3-openshift-service-ca\") pod \"a098155d-bc55-482b-8556-9c76db1fc5a3\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " Oct 30 02:53:17.399750 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.399414 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a098155d-bc55-482b-8556-9c76db1fc5a3-kube-api-access-rtkvl\" (UniqueName: \"kubernetes.io/projected/a098155d-bc55-482b-8556-9c76db1fc5a3-kube-api-access-rtkvl\") pod \"a098155d-bc55-482b-8556-9c76db1fc5a3\" (UID: \"a098155d-bc55-482b-8556-9c76db1fc5a3\") " Oct 30 02:53:17.399862 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.399837 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a098155d-bc55-482b-8556-9c76db1fc5a3-openshift-service-ca" pod "a098155d-bc55-482b-8556-9c76db1fc5a3" (UID: "a098155d-bc55-482b-8556-9c76db1fc5a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:53:17.399995 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.399974 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a098155d-bc55-482b-8556-9c76db1fc5a3-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:17.401857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.401827 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-maas-api-tls" pod "a098155d-bc55-482b-8556-9c76db1fc5a3" (UID: "a098155d-bc55-482b-8556-9c76db1fc5a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:53:17.402008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.401903 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-metrics-certs" pod "a098155d-bc55-482b-8556-9c76db1fc5a3" (UID: "a098155d-bc55-482b-8556-9c76db1fc5a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:53:17.402055 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.402027 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a098155d-bc55-482b-8556-9c76db1fc5a3-kube-api-access-rtkvl" pod "a098155d-bc55-482b-8556-9c76db1fc5a3" (UID: "a098155d-bc55-482b-8556-9c76db1fc5a3"). InnerVolumeSpecName "kube-api-access-rtkvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:53:17.500997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.500947 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:17.500997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.500989 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a098155d-bc55-482b-8556-9c76db1fc5a3-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:17.500997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:17.501000 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtkvl\" (UniqueName: \"kubernetes.io/projected/a098155d-bc55-482b-8556-9c76db1fc5a3-kube-api-access-rtkvl\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:17.629666 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:53:17.629623 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:53:18.036644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:18.036602 5035 generic.go:356] "Generic (PLEG): container finished" podID="a098155d-bc55-482b-8556-9c76db1fc5a3" containerID="6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c" exitCode=0 Oct 30 02:53:18.037216 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:18.036647 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" Oct 30 02:53:18.037216 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:18.036663 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" event={"ID":"a098155d-bc55-482b-8556-9c76db1fc5a3","Type":"ContainerDied","Data":"6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c"} Oct 30 02:53:18.037216 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:18.036728 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744" event={"ID":"a098155d-bc55-482b-8556-9c76db1fc5a3","Type":"ContainerDied","Data":"59f4b76014cf93acf10137ae223865a7d7f39e5f08057cd99afde50a1e9dca7e"} Oct 30 02:53:18.037216 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:18.036743 5035 scope.go:122] "RemoveContainer" containerID="6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c" Oct 30 02:53:18.049864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:18.049821 5035 scope.go:122] "RemoveContainer" containerID="6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c" Oct 30 02:53:18.050349 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:53:18.050312 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c\": container with ID starting with 6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c not found: ID does not exist" containerID="6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c" Oct 30 02:53:18.050420 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:18.050364 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c"} err="failed to get container status \"6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c\": rpc error: code = NotFound desc = could not find container \"6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c\": container with ID starting with 6157b30607a547bae6b36dae67a7ef07b7cc00a12a31756f1416563a25fe4c2c not found: ID does not exist" Oct 30 02:53:18.089815 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:18.089767 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744"] Oct 30 02:53:18.101887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:18.101851 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-88fa81-6b947c87b5-m6744"] Oct 30 02:53:19.511602 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:19.511502 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" podUID="0b1b01c4-c63b-432f-b5b3-887b02a209d4" containerName="payload-processing" containerID="cri-o://2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281" gracePeriod=25 Oct 30 02:53:19.543618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:19.543570 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" podUID="43ac2b09-8aae-4f5a-a842-b74e70d50e39" containerName="payload-pre-processing" containerID="cri-o://ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d" gracePeriod=25 Oct 30 02:53:19.641766 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:19.641730 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a098155d-bc55-482b-8556-9c76db1fc5a3" path="/var/lib/kubelet/pods/a098155d-bc55-482b-8556-9c76db1fc5a3/volumes" Oct 30 02:53:22.602960 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:22.602916 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" podUID="0b1b01c4-c63b-432f-b5b3-887b02a209d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:53:22.603597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:22.603125 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" podUID="43ac2b09-8aae-4f5a-a842-b74e70d50e39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:53:22.938229 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:22.937953 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c"] Oct 30 02:53:22.938426 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:22.938241 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" containerID="cri-o://1769d0b8ba261e9459c6779f31dc7796d27f61ff5f7d5d17bcbc47c9c71cdc7f" gracePeriod=30 Oct 30 02:53:28.103440 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.103398 5035 generic.go:356] "Generic (PLEG): container finished" podID="8e1d31d8-4a96-490e-98de-5b1926986894" containerID="1769d0b8ba261e9459c6779f31dc7796d27f61ff5f7d5d17bcbc47c9c71cdc7f" exitCode=0 Oct 30 02:53:28.103980 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.103468 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" event={"ID":"8e1d31d8-4a96-490e-98de-5b1926986894","Type":"ContainerDied","Data":"1769d0b8ba261e9459c6779f31dc7796d27f61ff5f7d5d17bcbc47c9c71cdc7f"} Oct 30 02:53:28.103980 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.103547 5035 scope.go:122] "RemoveContainer" containerID="9ddabd16d2c0e721c661987473d7bc6982b49657cc16159dd0b955c6ceef6509" Oct 30 02:53:28.216012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.215940 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:53:28.324283 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324242 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-istio-token\") pod \"8e1d31d8-4a96-490e-98de-5b1926986894\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " Oct 30 02:53:28.324482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324318 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-certs\") pod \"8e1d31d8-4a96-490e-98de-5b1926986894\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " Oct 30 02:53:28.324482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324355 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-socket\") pod \"8e1d31d8-4a96-490e-98de-5b1926986894\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " Oct 30 02:53:28.324482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324436 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-envoy\") pod \"8e1d31d8-4a96-490e-98de-5b1926986894\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " Oct 30 02:53:28.324707 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324560 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-kube-api-access-4vxgr\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-kube-api-access-4vxgr\") pod \"8e1d31d8-4a96-490e-98de-5b1926986894\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " Oct 30 02:53:28.324707 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324594 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-credential-socket\") pod \"8e1d31d8-4a96-490e-98de-5b1926986894\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " Oct 30 02:53:28.324707 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324647 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e1d31d8-4a96-490e-98de-5b1926986894-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d31d8-4a96-490e-98de-5b1926986894-istio-podinfo\") pod \"8e1d31d8-4a96-490e-98de-5b1926986894\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " Oct 30 02:53:28.324889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324665 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-certs" pod "8e1d31d8-4a96-490e-98de-5b1926986894" (UID: "8e1d31d8-4a96-490e-98de-5b1926986894"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:28.324889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324701 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-data\") pod \"8e1d31d8-4a96-490e-98de-5b1926986894\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " Oct 30 02:53:28.324889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324759 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-socket" pod "8e1d31d8-4a96-490e-98de-5b1926986894" (UID: "8e1d31d8-4a96-490e-98de-5b1926986894"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:28.324889 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324775 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1d31d8-4a96-490e-98de-5b1926986894-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d31d8-4a96-490e-98de-5b1926986894-istiod-ca-cert\") pod \"8e1d31d8-4a96-490e-98de-5b1926986894\" (UID: \"8e1d31d8-4a96-490e-98de-5b1926986894\") " Oct 30 02:53:28.325130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324934 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-data" pod "8e1d31d8-4a96-490e-98de-5b1926986894" (UID: "8e1d31d8-4a96-490e-98de-5b1926986894"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:28.325130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.324954 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-credential-socket" pod "8e1d31d8-4a96-490e-98de-5b1926986894" (UID: "8e1d31d8-4a96-490e-98de-5b1926986894"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:28.325274 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.325251 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1d31d8-4a96-490e-98de-5b1926986894-istiod-ca-cert" pod "8e1d31d8-4a96-490e-98de-5b1926986894" (UID: "8e1d31d8-4a96-490e-98de-5b1926986894"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:53:28.325391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.325372 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.325461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.325398 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1d31d8-4a96-490e-98de-5b1926986894-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.325461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.325412 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.325461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.325425 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.325461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.325440 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.326807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.326774 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-istio-token" pod "8e1d31d8-4a96-490e-98de-5b1926986894" (UID: "8e1d31d8-4a96-490e-98de-5b1926986894"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:53:28.326951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.326851 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-envoy" pod "8e1d31d8-4a96-490e-98de-5b1926986894" (UID: "8e1d31d8-4a96-490e-98de-5b1926986894"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:28.326951 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.326887 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-kube-api-access-4vxgr" pod "8e1d31d8-4a96-490e-98de-5b1926986894" (UID: "8e1d31d8-4a96-490e-98de-5b1926986894"). InnerVolumeSpecName "kube-api-access-4vxgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:53:28.327063 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.326951 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e1d31d8-4a96-490e-98de-5b1926986894-istio-podinfo" pod "8e1d31d8-4a96-490e-98de-5b1926986894" (UID: "8e1d31d8-4a96-490e-98de-5b1926986894"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:53:28.426041 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.426000 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vxgr\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-kube-api-access-4vxgr\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.426041 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.426037 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1d31d8-4a96-490e-98de-5b1926986894-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.426041 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.426050 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1d31d8-4a96-490e-98de-5b1926986894-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.426344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.426059 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1d31d8-4a96-490e-98de-5b1926986894-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.542235 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.542127 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" podUID="0b52547c-2763-4ed8-9960-da30f0b0a9d2" containerName="main" containerID="cri-o://685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93" gracePeriod=45 Oct 30 02:53:28.797069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.797003 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:53:28.932959 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.932920 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tls-certs\") pod \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " Oct 30 02:53:28.933202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.932993 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-dshm\") pod \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " Oct 30 02:53:28.933202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933084 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-home\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-home\") pod \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " Oct 30 02:53:28.933202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933157 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b52547c-2763-4ed8-9960-da30f0b0a9d2-kube-api-access-v6fcn\" (UniqueName: \"kubernetes.io/projected/0b52547c-2763-4ed8-9960-da30f0b0a9d2-kube-api-access-v6fcn\") pod \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " Oct 30 02:53:28.933202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933194 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tmp-dir\") pod \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " Oct 30 02:53:28.933451 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933263 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-model-cache\") pod \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\" (UID: \"0b52547c-2763-4ed8-9960-da30f0b0a9d2\") " Oct 30 02:53:28.933451 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933381 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-home" pod "0b52547c-2763-4ed8-9960-da30f0b0a9d2" (UID: "0b52547c-2763-4ed8-9960-da30f0b0a9d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:28.933587 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933516 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tmp-dir" pod "0b52547c-2763-4ed8-9960-da30f0b0a9d2" (UID: "0b52547c-2763-4ed8-9960-da30f0b0a9d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:28.933649 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933594 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-model-cache" pod "0b52547c-2763-4ed8-9960-da30f0b0a9d2" (UID: "0b52547c-2763-4ed8-9960-da30f0b0a9d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:28.933806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933788 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.933875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933808 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.933875 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.933816 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:28.935490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.935459 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tls-certs" pod "0b52547c-2763-4ed8-9960-da30f0b0a9d2" (UID: "0b52547c-2763-4ed8-9960-da30f0b0a9d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:53:28.935490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.935464 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-dshm" pod "0b52547c-2763-4ed8-9960-da30f0b0a9d2" (UID: "0b52547c-2763-4ed8-9960-da30f0b0a9d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:28.935680 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:28.935489 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b52547c-2763-4ed8-9960-da30f0b0a9d2-kube-api-access-v6fcn" pod "0b52547c-2763-4ed8-9960-da30f0b0a9d2" (UID: "0b52547c-2763-4ed8-9960-da30f0b0a9d2"). InnerVolumeSpecName "kube-api-access-v6fcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:53:29.034771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.034726 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6fcn\" (UniqueName: \"kubernetes.io/projected/0b52547c-2763-4ed8-9960-da30f0b0a9d2-kube-api-access-v6fcn\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:29.034771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.034766 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b52547c-2763-4ed8-9960-da30f0b0a9d2-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:29.034771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.034776 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b52547c-2763-4ed8-9960-da30f0b0a9d2-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:29.110495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.110400 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" Oct 30 02:53:29.110495 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.110445 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c" event={"ID":"8e1d31d8-4a96-490e-98de-5b1926986894","Type":"ContainerDied","Data":"99c72df60aaa61d7d42a9c25c70b8ac9e2ef6d274a5823c3245cfeb902e5cb5b"} Oct 30 02:53:29.111062 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.110504 5035 scope.go:122] "RemoveContainer" containerID="1769d0b8ba261e9459c6779f31dc7796d27f61ff5f7d5d17bcbc47c9c71cdc7f" Oct 30 02:53:29.111860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.111835 5035 generic.go:356] "Generic (PLEG): container finished" podID="0b52547c-2763-4ed8-9960-da30f0b0a9d2" containerID="685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93" exitCode=0 Oct 30 02:53:29.111945 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.111880 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" Oct 30 02:53:29.111945 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.111926 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" event={"ID":"0b52547c-2763-4ed8-9960-da30f0b0a9d2","Type":"ContainerDied","Data":"685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93"} Oct 30 02:53:29.112036 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.111972 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" event={"ID":"0b52547c-2763-4ed8-9960-da30f0b0a9d2","Type":"ContainerDied","Data":"939c10644cdb4e814eb947a0a550adda9759ecd9dda6cd27a1ba2c1a1797fec9"} Oct 30 02:53:29.120968 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.120939 5035 scope.go:122] "RemoveContainer" containerID="685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93" Oct 30 02:53:29.130950 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.130924 5035 scope.go:122] "RemoveContainer" containerID="685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93" Oct 30 02:53:29.131257 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:53:29.131234 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93\": container with ID starting with 685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93 not found: ID does not exist" containerID="685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93" Oct 30 02:53:29.131300 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.131270 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93"} err="failed to get container status \"685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93\": rpc error: code = NotFound desc = could not find container \"685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93\": container with ID starting with 685ec6667a3a8f45642fccf28f3b75b89fa1046f217bba7853bb0e8c3e9d3c93 not found: ID does not exist" Oct 30 02:53:29.162737 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.162700 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg"] Oct 30 02:53:29.167183 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.167153 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg"] Oct 30 02:53:29.202143 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.202105 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c"] Oct 30 02:53:29.206650 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.206612 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88fa81-openshift-default-b74b6f697-48r2c"] Oct 30 02:53:29.642750 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.642716 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b52547c-2763-4ed8-9960-da30f0b0a9d2" path="/var/lib/kubelet/pods/0b52547c-2763-4ed8-9960-da30f0b0a9d2/volumes" Oct 30 02:53:29.643182 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.643168 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" path="/var/lib/kubelet/pods/8e1d31d8-4a96-490e-98de-5b1926986894/volumes" Oct 30 02:53:29.785315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:29.785268 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-88fa81/simulator-single-main-88fa81-kserve-69659b4b8d-hxzhg" podUID="0b52547c-2763-4ed8-9960-da30f0b0a9d2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.22:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:53:32.626557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:32.624648 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" podUID="43ac2b09-8aae-4f5a-a842-b74e70d50e39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:53:32.626557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:32.625001 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" podUID="0b1b01c4-c63b-432f-b5b3-887b02a209d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:53:39.810117 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.810077 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:53:39.839303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.839271 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:53:39.980371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.980330 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ac2b09-8aae-4f5a-a842-b74e70d50e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ac2b09-8aae-4f5a-a842-b74e70d50e39-plugins-config-volume\") pod \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\" (UID: \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\") " Oct 30 02:53:39.980371 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.980372 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1b01c4-c63b-432f-b5b3-887b02a209d4-kube-api-access-vgl6l\" (UniqueName: \"kubernetes.io/projected/0b1b01c4-c63b-432f-b5b3-887b02a209d4-kube-api-access-vgl6l\") pod \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\" (UID: \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\") " Oct 30 02:53:39.980693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.980425 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ac2b09-8aae-4f5a-a842-b74e70d50e39-kube-api-access-b9skr\" (UniqueName: \"kubernetes.io/projected/43ac2b09-8aae-4f5a-a842-b74e70d50e39-kube-api-access-b9skr\") pod \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\" (UID: \"43ac2b09-8aae-4f5a-a842-b74e70d50e39\") " Oct 30 02:53:39.980693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.980620 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b1b01c4-c63b-432f-b5b3-887b02a209d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b01c4-c63b-432f-b5b3-887b02a209d4-plugins-config-volume\") pod \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\" (UID: \"0b1b01c4-c63b-432f-b5b3-887b02a209d4\") " Oct 30 02:53:39.980805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.980774 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ac2b09-8aae-4f5a-a842-b74e70d50e39-plugins-config-volume" pod "43ac2b09-8aae-4f5a-a842-b74e70d50e39" (UID: "43ac2b09-8aae-4f5a-a842-b74e70d50e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:53:39.981021 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.981000 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1b01c4-c63b-432f-b5b3-887b02a209d4-plugins-config-volume" pod "0b1b01c4-c63b-432f-b5b3-887b02a209d4" (UID: "0b1b01c4-c63b-432f-b5b3-887b02a209d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:53:39.981223 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.981207 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b01c4-c63b-432f-b5b3-887b02a209d4-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:39.981273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.981230 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ac2b09-8aae-4f5a-a842-b74e70d50e39-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:39.982724 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.982698 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1b01c4-c63b-432f-b5b3-887b02a209d4-kube-api-access-vgl6l" pod "0b1b01c4-c63b-432f-b5b3-887b02a209d4" (UID: "0b1b01c4-c63b-432f-b5b3-887b02a209d4"). InnerVolumeSpecName "kube-api-access-vgl6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:53:39.982724 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:39.982712 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ac2b09-8aae-4f5a-a842-b74e70d50e39-kube-api-access-b9skr" pod "43ac2b09-8aae-4f5a-a842-b74e70d50e39" (UID: "43ac2b09-8aae-4f5a-a842-b74e70d50e39"). InnerVolumeSpecName "kube-api-access-b9skr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:53:40.082375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.082277 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgl6l\" (UniqueName: \"kubernetes.io/projected/0b1b01c4-c63b-432f-b5b3-887b02a209d4-kube-api-access-vgl6l\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:40.082375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.082315 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9skr\" (UniqueName: \"kubernetes.io/projected/43ac2b09-8aae-4f5a-a842-b74e70d50e39-kube-api-access-b9skr\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:53:40.344486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.344384 5035 generic.go:356] "Generic (PLEG): container finished" podID="0b1b01c4-c63b-432f-b5b3-887b02a209d4" containerID="2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281" exitCode=0 Oct 30 02:53:40.344486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.344435 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" Oct 30 02:53:40.344486 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.344460 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" event={"ID":"0b1b01c4-c63b-432f-b5b3-887b02a209d4","Type":"ContainerDied","Data":"2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281"} Oct 30 02:53:40.344815 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.344510 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8" event={"ID":"0b1b01c4-c63b-432f-b5b3-887b02a209d4","Type":"ContainerDied","Data":"b0248ce507629c557435d8f8b545635f0dec86347bb6e675fa7a3f35e9b51260"} Oct 30 02:53:40.344815 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.344548 5035 scope.go:122] "RemoveContainer" containerID="2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281" Oct 30 02:53:40.346122 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.346099 5035 generic.go:356] "Generic (PLEG): container finished" podID="43ac2b09-8aae-4f5a-a842-b74e70d50e39" containerID="ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d" exitCode=0 Oct 30 02:53:40.346277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.346152 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" Oct 30 02:53:40.346277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.346178 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" event={"ID":"43ac2b09-8aae-4f5a-a842-b74e70d50e39","Type":"ContainerDied","Data":"ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d"} Oct 30 02:53:40.346277 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.346224 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj" event={"ID":"43ac2b09-8aae-4f5a-a842-b74e70d50e39","Type":"ContainerDied","Data":"ca4c83af652e52066da4bb11fcbb8a9848196c6885de81ffc4a700638c9135df"} Oct 30 02:53:40.354303 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.354273 5035 scope.go:122] "RemoveContainer" containerID="2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281" Oct 30 02:53:40.354642 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:53:40.354617 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281\": container with ID starting with 2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281 not found: ID does not exist" containerID="2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281" Oct 30 02:53:40.354847 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.354653 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281"} err="failed to get container status \"2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281\": rpc error: code = NotFound desc = could not find container \"2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281\": container with ID starting with 2ce683ec98e629665f0d1da7b895e4973b8f27eb727091205d8b45e8ab822281 not found: ID does not exist" Oct 30 02:53:40.354847 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.354678 5035 scope.go:122] "RemoveContainer" containerID="ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d" Oct 30 02:53:40.364441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.364418 5035 scope.go:122] "RemoveContainer" containerID="ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d" Oct 30 02:53:40.364777 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:53:40.364749 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d\": container with ID starting with ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d not found: ID does not exist" containerID="ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d" Oct 30 02:53:40.364887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.364785 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d"} err="failed to get container status \"ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d\": rpc error: code = NotFound desc = could not find container \"ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d\": container with ID starting with ffd49f1636fa200a5051c4cc0d791c647f22955ebff7f0e3d23e0b3841612d4d not found: ID does not exist" Oct 30 02:53:40.396293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.396259 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj"] Oct 30 02:53:40.398913 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.398888 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-88fa81-77b89jt6pj"] Oct 30 02:53:40.442881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.442841 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8"] Oct 30 02:53:40.446912 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:40.446871 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-88fa81-697c96c4d9bvd8"] Oct 30 02:53:41.641986 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:41.641949 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1b01c4-c63b-432f-b5b3-887b02a209d4" path="/var/lib/kubelet/pods/0b1b01c4-c63b-432f-b5b3-887b02a209d4/volumes" Oct 30 02:53:41.642391 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:41.642351 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ac2b09-8aae-4f5a-a842-b74e70d50e39" path="/var/lib/kubelet/pods/43ac2b09-8aae-4f5a-a842-b74e70d50e39/volumes" Oct 30 02:53:44.080255 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080215 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5"] Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080771 5035 cpu_manager.go:414] "containerMap: removing container" podUID="a098155d-bc55-482b-8556-9c76db1fc5a3" containerName="maas-api" Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080791 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a098155d-bc55-482b-8556-9c76db1fc5a3" containerName="maas-api" Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080820 5035 cpu_manager.go:414] "containerMap: removing container" podUID="43ac2b09-8aae-4f5a-a842-b74e70d50e39" containerName="payload-pre-processing" Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080825 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ac2b09-8aae-4f5a-a842-b74e70d50e39" containerName="payload-pre-processing" Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080839 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080844 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080853 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0b1b01c4-c63b-432f-b5b3-887b02a209d4" containerName="payload-processing" Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080860 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1b01c4-c63b-432f-b5b3-887b02a209d4" containerName="payload-processing" Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080869 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" Oct 30 02:53:44.080868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080877 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" Oct 30 02:53:44.081360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080884 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0b52547c-2763-4ed8-9960-da30f0b0a9d2" containerName="main" Oct 30 02:53:44.081360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080889 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b52547c-2763-4ed8-9960-da30f0b0a9d2" containerName="main" Oct 30 02:53:44.081360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080950 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" Oct 30 02:53:44.081360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080958 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1b01c4-c63b-432f-b5b3-887b02a209d4" containerName="payload-processing" Oct 30 02:53:44.081360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080965 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1d31d8-4a96-490e-98de-5b1926986894" containerName="istio-proxy" Oct 30 02:53:44.081360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080973 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b52547c-2763-4ed8-9960-da30f0b0a9d2" containerName="main" Oct 30 02:53:44.081360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080981 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="a098155d-bc55-482b-8556-9c76db1fc5a3" containerName="maas-api" Oct 30 02:53:44.081360 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.080991 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ac2b09-8aae-4f5a-a842-b74e70d50e39" containerName="payload-pre-processing" Oct 30 02:53:44.085902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.085872 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.105735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.105698 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5"] Oct 30 02:53:44.114009 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.113967 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-50b514ff-kserve-self-signed-certs\"" Oct 30 02:53:44.120595 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.120553 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tls-certs\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.120795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.120616 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-dshm\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.120856 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.120786 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-model-cache\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.120856 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.120834 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-home\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.120958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.120875 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bps57\" (UniqueName: \"kubernetes.io/projected/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-kube-api-access-bps57\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.121031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.121006 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tmp-dir\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.189888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.189857 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:53:44.201668 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.201640 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:53:44.206266 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.206239 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:53:44.221865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.221826 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tls-certs\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.221865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.221869 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-dshm\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.222127 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.221926 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-model-cache\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.222127 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.221945 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-home\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.222127 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.221968 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bps57\" (UniqueName: \"kubernetes.io/projected/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-kube-api-access-bps57\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.222127 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.221993 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tmp-dir\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.222406 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.222381 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tmp-dir\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.222482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.222456 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-home\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.222574 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.222508 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-model-cache\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.224270 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.224250 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-dshm\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.224454 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.224435 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tls-certs\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.245144 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.245105 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bps57\" (UniqueName: \"kubernetes.io/projected/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-kube-api-access-bps57\") pod \"e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.291415 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.291377 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:53:44.309768 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.309739 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:53:44.314320 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.314295 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:53:44.407130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.407000 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:44.725477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:44.725444 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5"] Oct 30 02:53:44.727282 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:53:44.727252 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18a11ef4_1a3d_4d35_ac01_68ad67aa0586.slice/crio-f655c1f3df4fce492187c0148c72647496e840a9c544fe2d88bda2044ff94a74 WatchSource:0}: Error finding container f655c1f3df4fce492187c0148c72647496e840a9c544fe2d88bda2044ff94a74: Status 404 returned error can't find the container with id f655c1f3df4fce492187c0148c72647496e840a9c544fe2d88bda2044ff94a74 Oct 30 02:53:45.019256 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:45.019125 5035 scope.go:122] "RemoveContainer" containerID="472826932a5b254360637ef8de38f8ecd616873ace612b8936dfa19562e9b2ec" Oct 30 02:53:45.029344 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:45.029320 5035 scope.go:122] "RemoveContainer" containerID="fd747100c7c28d7edea9be3077fb4526a223fb2b77e58ac645a51f0bcc2027f8" Oct 30 02:53:45.038618 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:45.038595 5035 scope.go:122] "RemoveContainer" containerID="097a77af8f4183a9eabadc34f713b01fec3af0a586301ac92c5b5284a94b2ab7" Oct 30 02:53:45.050652 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:45.050627 5035 scope.go:122] "RemoveContainer" containerID="926e9e3ad2eefec02e12409a178b95c50b5c76bd73e90e8065c7e71d3c2ca1ba" Oct 30 02:53:45.059991 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:45.059963 5035 scope.go:122] "RemoveContainer" containerID="6df51f5bc9da0d440c4f45ac0c0bdc75dc3894881906600acbfae1e8c2cf4b7d" Oct 30 02:53:45.383165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:45.383069 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" event={"ID":"18a11ef4-1a3d-4d35-ac01-68ad67aa0586","Type":"ContainerStarted","Data":"b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a"} Oct 30 02:53:45.383165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:45.383123 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" event={"ID":"18a11ef4-1a3d-4d35-ac01-68ad67aa0586","Type":"ContainerStarted","Data":"f655c1f3df4fce492187c0148c72647496e840a9c544fe2d88bda2044ff94a74"} Oct 30 02:53:45.407726 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:45.407688 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:45.428033 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:45.427968 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" podStartSLOduration=1.42795114 podStartE2EDuration="1.42795114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:45.42517898 +0000 UTC m=+5103.928111226" watchObservedRunningTime="2026-10-30 02:53:45.42795114 +0000 UTC m=+5103.930883359" Oct 30 02:53:47.657551 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:53:47.657238 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:53:54.407976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:54.407930 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:54.421138 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:54.421100 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:53:54.521178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:53:54.521143 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:54:06.112330 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.112283 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b"] Oct 30 02:54:06.116292 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.116262 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.145262 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.145219 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:54:06.145488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.145308 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:54:06.145488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.145355 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:54:06.145488 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.145373 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:54:06.148903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.148864 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b"] Oct 30 02:54:06.163032 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.162990 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4n9b\" (UniqueName: \"kubernetes.io/projected/0e2a7029-0880-48c1-8849-d744364ccc19-kube-api-access-t4n9b\") pod \"maas-api-revoke-sub-452b6d7b-szm8b\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.163221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.163145 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-szm8b\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.163297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.163228 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-szm8b\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.264198 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.264154 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-szm8b\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.264401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.264224 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-szm8b\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.264401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.264304 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4n9b\" (UniqueName: \"kubernetes.io/projected/0e2a7029-0880-48c1-8849-d744364ccc19-kube-api-access-t4n9b\") pod \"maas-api-revoke-sub-452b6d7b-szm8b\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.264998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.264962 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-szm8b\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.266743 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.266711 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-szm8b\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.288008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.287970 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4n9b\" (UniqueName: \"kubernetes.io/projected/0e2a7029-0880-48c1-8849-d744364ccc19-kube-api-access-t4n9b\") pod \"maas-api-revoke-sub-452b6d7b-szm8b\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.430075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.429984 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:06.739903 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:06.739872 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b"] Oct 30 02:54:06.741945 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:54:06.741907 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2a7029_0880_48c1_8849_d744364ccc19.slice/crio-354b7f28879b039dc62bf6fd2e1f365197a4184b8fa6dc9eb8598f77b736d92a WatchSource:0}: Error finding container 354b7f28879b039dc62bf6fd2e1f365197a4184b8fa6dc9eb8598f77b736d92a: Status 404 returned error can't find the container with id 354b7f28879b039dc62bf6fd2e1f365197a4184b8fa6dc9eb8598f77b736d92a Oct 30 02:54:07.034210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:07.034054 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx"] Oct 30 02:54:07.037038 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:07.037009 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lhsfx"] Oct 30 02:54:07.586934 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:07.586890 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" event={"ID":"0e2a7029-0880-48c1-8849-d744364ccc19","Type":"ContainerStarted","Data":"927d11d9b11626b7bd6aa04be65a4f79b77dfb7aa50f2f04a36be28a0e51a524"} Oct 30 02:54:07.587398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:07.586962 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" event={"ID":"0e2a7029-0880-48c1-8849-d744364ccc19","Type":"ContainerStarted","Data":"354b7f28879b039dc62bf6fd2e1f365197a4184b8fa6dc9eb8598f77b736d92a"} Oct 30 02:54:07.629593 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:07.629488 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" podStartSLOduration=1.629471659 podStartE2EDuration="1.629471659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:07.626836829 +0000 UTC m=+5126.129769050" watchObservedRunningTime="2026-10-30 02:54:07.629471659 +0000 UTC m=+5126.132403934" Oct 30 02:54:07.642315 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:07.642278 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae9c6804-a967-446b-ae0b-3030b03909b7" path="/var/lib/kubelet/pods/ae9c6804-a967-446b-ae0b-3030b03909b7/volumes" Oct 30 02:54:08.598479 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:08.598423 5035 generic.go:356] "Generic (PLEG): container finished" podID="0e2a7029-0880-48c1-8849-d744364ccc19" containerID="927d11d9b11626b7bd6aa04be65a4f79b77dfb7aa50f2f04a36be28a0e51a524" exitCode=0 Oct 30 02:54:08.599328 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:08.598515 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" event={"ID":"0e2a7029-0880-48c1-8849-d744364ccc19","Type":"ContainerDied","Data":"927d11d9b11626b7bd6aa04be65a4f79b77dfb7aa50f2f04a36be28a0e51a524"} Oct 30 02:54:10.570555 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.561871 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:10.604560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.585201 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-xks-ca\") pod \"0e2a7029-0880-48c1-8849-d744364ccc19\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " Oct 30 02:54:10.604560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.585821 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-openshift-ca\") pod \"0e2a7029-0880-48c1-8849-d744364ccc19\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " Oct 30 02:54:10.604560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.586206 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-openshift-ca" pod "0e2a7029-0880-48c1-8849-d744364ccc19" (UID: "0e2a7029-0880-48c1-8849-d744364ccc19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:54:10.604560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.586431 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2a7029-0880-48c1-8849-d744364ccc19-kube-api-access-t4n9b\" (UniqueName: \"kubernetes.io/projected/0e2a7029-0880-48c1-8849-d744364ccc19-kube-api-access-t4n9b\") pod \"0e2a7029-0880-48c1-8849-d744364ccc19\" (UID: \"0e2a7029-0880-48c1-8849-d744364ccc19\") " Oct 30 02:54:10.604560 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.587146 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:10.642693 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.641189 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-xks-ca" pod "0e2a7029-0880-48c1-8849-d744364ccc19" (UID: "0e2a7029-0880-48c1-8849-d744364ccc19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:54:10.681155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.680986 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2a7029-0880-48c1-8849-d744364ccc19-kube-api-access-t4n9b" pod "0e2a7029-0880-48c1-8849-d744364ccc19" (UID: "0e2a7029-0880-48c1-8849-d744364ccc19"). InnerVolumeSpecName "kube-api-access-t4n9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:54:10.718648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.716423 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4n9b\" (UniqueName: \"kubernetes.io/projected/0e2a7029-0880-48c1-8849-d744364ccc19-kube-api-access-t4n9b\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:10.718648 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.716475 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e2a7029-0880-48c1-8849-d744364ccc19-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:10.960795 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.960745 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" event={"ID":"0e2a7029-0880-48c1-8849-d744364ccc19","Type":"ContainerDied","Data":"354b7f28879b039dc62bf6fd2e1f365197a4184b8fa6dc9eb8598f77b736d92a"} Oct 30 02:54:10.969759 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.969657 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="354b7f28879b039dc62bf6fd2e1f365197a4184b8fa6dc9eb8598f77b736d92a" Oct 30 02:54:10.973628 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:10.964886 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b" Oct 30 02:54:17.629641 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:54:17.629589 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:54:23.371372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:23.371329 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5"] Oct 30 02:54:27.038810 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.038767 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv"] Oct 30 02:54:27.039500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.039472 5035 cpu_manager.go:414] "containerMap: removing container" podUID="0e2a7029-0880-48c1-8849-d744364ccc19" containerName="revoke-keys" Oct 30 02:54:27.039500 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.039498 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2a7029-0880-48c1-8849-d744364ccc19" containerName="revoke-keys" Oct 30 02:54:27.039722 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.039703 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2a7029-0880-48c1-8849-d744364ccc19" containerName="revoke-keys" Oct 30 02:54:27.043683 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.043650 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.069997 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.069943 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv"] Oct 30 02:54:27.078808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.078421 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:54:27.078808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.078501 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:54:27.078808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.078676 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:54:27.078808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.078754 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:54:27.099383 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.099335 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqp67\" (UniqueName: \"kubernetes.io/projected/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-kube-api-access-nqp67\") pod \"maas-api-revoke-sub-3e31a506-rpmfv\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.099860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.099828 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rpmfv\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.100006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.099956 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rpmfv\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.200833 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.200778 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rpmfv\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.201071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.200910 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqp67\" (UniqueName: \"kubernetes.io/projected/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-kube-api-access-nqp67\") pod \"maas-api-revoke-sub-3e31a506-rpmfv\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.201071 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.201010 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rpmfv\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.201888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.201853 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rpmfv\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.204843 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.204811 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rpmfv\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.226971 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.226929 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqp67\" (UniqueName: \"kubernetes.io/projected/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-kube-api-access-nqp67\") pod \"maas-api-revoke-sub-3e31a506-rpmfv\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.368792 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.368670 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:27.766569 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:27.762457 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv"] Oct 30 02:54:27.769480 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:54:27.769442 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba7b1ea_b0fc_4ca0_8a6b_6a89652ed658.slice/crio-dc3e9bebd1b905069d2bb578d720f41eb32406940ff741d9878c53c7011f0a65 WatchSource:0}: Error finding container dc3e9bebd1b905069d2bb578d720f41eb32406940ff741d9878c53c7011f0a65: Status 404 returned error can't find the container with id dc3e9bebd1b905069d2bb578d720f41eb32406940ff741d9878c53c7011f0a65 Oct 30 02:54:28.260878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:28.260828 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" event={"ID":"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658","Type":"ContainerStarted","Data":"dc3e9bebd1b905069d2bb578d720f41eb32406940ff741d9878c53c7011f0a65"} Oct 30 02:54:29.306112 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:29.306051 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" event={"ID":"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658","Type":"ContainerStarted","Data":"4572e4c9bb6550c0f5b5e3c09f9ddc4bdc1a5b9fb743feda31c4757083cd3716"} Oct 30 02:54:29.389647 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:29.389569 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" podStartSLOduration=2.3895451039999998 podStartE2EDuration="2.389545104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:29.388889676 +0000 UTC m=+5147.891821897" watchObservedRunningTime="2026-10-30 02:54:29.389545104 +0000 UTC m=+5147.892477339" Oct 30 02:54:30.313924 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:30.313880 5035 generic.go:356] "Generic (PLEG): container finished" podID="5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658" containerID="4572e4c9bb6550c0f5b5e3c09f9ddc4bdc1a5b9fb743feda31c4757083cd3716" exitCode=0 Oct 30 02:54:30.314376 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:30.313968 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" event={"ID":"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658","Type":"ContainerDied","Data":"4572e4c9bb6550c0f5b5e3c09f9ddc4bdc1a5b9fb743feda31c4757083cd3716"} Oct 30 02:54:31.468557 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.468500 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:31.574012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.573969 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-kube-api-access-nqp67\" (UniqueName: \"kubernetes.io/projected/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-kube-api-access-nqp67\") pod \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " Oct 30 02:54:31.574228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.574038 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-xks-ca\") pod \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " Oct 30 02:54:31.574228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.574117 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-openshift-ca\") pod \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\" (UID: \"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658\") " Oct 30 02:54:31.574578 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.574508 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-openshift-ca" pod "5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658" (UID: "5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:54:31.576249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.576223 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-xks-ca" pod "5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658" (UID: "5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:54:31.576249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.576233 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-kube-api-access-nqp67" pod "5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658" (UID: "5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658"). InnerVolumeSpecName "kube-api-access-nqp67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:54:31.675790 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.675747 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqp67\" (UniqueName: \"kubernetes.io/projected/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-kube-api-access-nqp67\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:31.675790 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.675785 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:31.675790 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:31.675794 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:32.326998 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:32.326961 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" Oct 30 02:54:32.327201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:32.326960 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv" event={"ID":"5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658","Type":"ContainerDied","Data":"dc3e9bebd1b905069d2bb578d720f41eb32406940ff741d9878c53c7011f0a65"} Oct 30 02:54:32.327201 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:32.327092 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc3e9bebd1b905069d2bb578d720f41eb32406940ff741d9878c53c7011f0a65" Oct 30 02:54:38.394097 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.394052 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" podUID="18a11ef4-1a3d-4d35-ac01-68ad67aa0586" containerName="main" containerID="cri-o://b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a" gracePeriod=45 Oct 30 02:54:38.510323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.510280 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" podUID="18a11ef4-1a3d-4d35-ac01-68ad67aa0586" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/ready\": dial tcp 10.132.1.23:8000: connect: connection refused" Oct 30 02:54:38.654477 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.654397 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:54:38.755637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.755556 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-kube-api-access-bps57\" (UniqueName: \"kubernetes.io/projected/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-kube-api-access-bps57\") pod \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " Oct 30 02:54:38.755637 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.755648 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-dshm\") pod \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " Oct 30 02:54:38.755957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.755710 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-model-cache\") pod \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " Oct 30 02:54:38.755957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.755742 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tls-certs\" (UniqueName: \"kubernetes.io/secret/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tls-certs\") pod \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " Oct 30 02:54:38.755957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.755805 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tmp-dir\") pod \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " Oct 30 02:54:38.755957 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.755933 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-home\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-home\") pod \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\" (UID: \"18a11ef4-1a3d-4d35-ac01-68ad67aa0586\") " Oct 30 02:54:38.756789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.756579 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-home" pod "18a11ef4-1a3d-4d35-ac01-68ad67aa0586" (UID: "18a11ef4-1a3d-4d35-ac01-68ad67aa0586"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:54:38.756789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.756665 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-model-cache" pod "18a11ef4-1a3d-4d35-ac01-68ad67aa0586" (UID: "18a11ef4-1a3d-4d35-ac01-68ad67aa0586"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:54:38.757052 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.756889 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tmp-dir" pod "18a11ef4-1a3d-4d35-ac01-68ad67aa0586" (UID: "18a11ef4-1a3d-4d35-ac01-68ad67aa0586"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:54:38.758865 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.758835 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tls-certs" pod "18a11ef4-1a3d-4d35-ac01-68ad67aa0586" (UID: "18a11ef4-1a3d-4d35-ac01-68ad67aa0586"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:54:38.759658 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.759627 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-dshm" pod "18a11ef4-1a3d-4d35-ac01-68ad67aa0586" (UID: "18a11ef4-1a3d-4d35-ac01-68ad67aa0586"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:54:38.759776 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.759759 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-kube-api-access-bps57" pod "18a11ef4-1a3d-4d35-ac01-68ad67aa0586" (UID: "18a11ef4-1a3d-4d35-ac01-68ad67aa0586"). InnerVolumeSpecName "kube-api-access-bps57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:54:38.858577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.857347 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:38.858577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.857401 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:38.858577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.857419 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bps57\" (UniqueName: \"kubernetes.io/projected/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-kube-api-access-bps57\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:38.858577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.857433 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:38.858577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.857448 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:38.858577 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:38.857460 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18a11ef4-1a3d-4d35-ac01-68ad67aa0586-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:39.389553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.389254 5035 generic.go:356] "Generic (PLEG): container finished" podID="18a11ef4-1a3d-4d35-ac01-68ad67aa0586" containerID="b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a" exitCode=0 Oct 30 02:54:39.389553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.389345 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" event={"ID":"18a11ef4-1a3d-4d35-ac01-68ad67aa0586","Type":"ContainerDied","Data":"b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a"} Oct 30 02:54:39.389553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.389387 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" event={"ID":"18a11ef4-1a3d-4d35-ac01-68ad67aa0586","Type":"ContainerDied","Data":"f655c1f3df4fce492187c0148c72647496e840a9c544fe2d88bda2044ff94a74"} Oct 30 02:54:39.389553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.389412 5035 scope.go:122] "RemoveContainer" containerID="b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a" Oct 30 02:54:39.390001 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.389632 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5" Oct 30 02:54:39.461851 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.459737 5035 scope.go:122] "RemoveContainer" containerID="b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a" Oct 30 02:54:39.462864 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:54:39.462710 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a\": container with ID starting with b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a not found: ID does not exist" containerID="b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a" Oct 30 02:54:39.462864 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.462762 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a"} err="failed to get container status \"b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a\": rpc error: code = NotFound desc = could not find container \"b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a\": container with ID starting with b4fe7d49d70899a683a57ec2ee7e50484e6070045b146f43612e8e0138991e1a not found: ID does not exist" Oct 30 02:54:39.486629 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.486583 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5"] Oct 30 02:54:39.503942 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.503848 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-50b514ff-kserve-8448dcb4f7-zk4k5"] Oct 30 02:54:39.656681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:39.655466 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a11ef4-1a3d-4d35-ac01-68ad67aa0586" path="/var/lib/kubelet/pods/18a11ef4-1a3d-4d35-ac01-68ad67aa0586/volumes" Oct 30 02:54:45.125883 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.125839 5035 scope.go:122] "RemoveContainer" containerID="51b3316aa023f4aea10583199610a5d391cbdc0fa7f828b3e9f936921420ec21" Oct 30 02:54:45.823621 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.823577 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5"] Oct 30 02:54:45.824099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.824081 5035 cpu_manager.go:414] "containerMap: removing container" podUID="18a11ef4-1a3d-4d35-ac01-68ad67aa0586" containerName="main" Oct 30 02:54:45.824155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.824102 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a11ef4-1a3d-4d35-ac01-68ad67aa0586" containerName="main" Oct 30 02:54:45.824155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.824126 5035 cpu_manager.go:414] "containerMap: removing container" podUID="5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658" containerName="revoke-keys" Oct 30 02:54:45.824155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.824132 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658" containerName="revoke-keys" Oct 30 02:54:45.824254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.824198 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a11ef4-1a3d-4d35-ac01-68ad67aa0586" containerName="main" Oct 30 02:54:45.824254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.824210 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658" containerName="revoke-keys" Oct 30 02:54:45.829080 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.829058 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:45.846574 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.846521 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:54:45.851040 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.851010 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5"] Oct 30 02:54:45.858011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.857978 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:54:45.858011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.857992 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:54:45.858323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.858025 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:54:45.915611 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.915487 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7db290-tszn5\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:45.915611 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.915571 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tqwq\" (UniqueName: \"kubernetes.io/projected/e42a265e-d25e-41ae-82f5-4e1880a0ca29-kube-api-access-6tqwq\") pod \"maas-api-revoke-sub-6c7db290-tszn5\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:45.915837 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:45.915692 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7db290-tszn5\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:46.017083 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.017041 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tqwq\" (UniqueName: \"kubernetes.io/projected/e42a265e-d25e-41ae-82f5-4e1880a0ca29-kube-api-access-6tqwq\") pod \"maas-api-revoke-sub-6c7db290-tszn5\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:46.017295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.017112 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7db290-tszn5\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:46.017295 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.017210 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7db290-tszn5\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:46.018005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.017974 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7db290-tszn5\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:46.019645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.019626 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7db290-tszn5\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:46.032439 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.032402 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5"] Oct 30 02:54:46.034979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.034947 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-knvh5"] Oct 30 02:54:46.040190 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.040159 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tqwq\" (UniqueName: \"kubernetes.io/projected/e42a265e-d25e-41ae-82f5-4e1880a0ca29-kube-api-access-6tqwq\") pod \"maas-api-revoke-sub-6c7db290-tszn5\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:46.143382 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.143277 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:46.446736 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.446709 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5"] Oct 30 02:54:46.448831 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:54:46.448798 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42a265e_d25e_41ae_82f5_4e1880a0ca29.slice/crio-fc12fddf0af77275a502c1bb6520a18349b126294f780548a98292a165cd2cd2 WatchSource:0}: Error finding container fc12fddf0af77275a502c1bb6520a18349b126294f780548a98292a165cd2cd2: Status 404 returned error can't find the container with id fc12fddf0af77275a502c1bb6520a18349b126294f780548a98292a165cd2cd2 Oct 30 02:54:46.473013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:46.472977 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" event={"ID":"e42a265e-d25e-41ae-82f5-4e1880a0ca29","Type":"ContainerStarted","Data":"fc12fddf0af77275a502c1bb6520a18349b126294f780548a98292a165cd2cd2"} Oct 30 02:54:47.489536 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:47.489494 5035 generic.go:356] "Generic (PLEG): container finished" podID="e42a265e-d25e-41ae-82f5-4e1880a0ca29" containerID="366006eba7b02e7b5b7829de0f2984582a721b1ddbfe6ba644e0b76967a420a8" exitCode=0 Oct 30 02:54:47.490048 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:47.489588 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" event={"ID":"e42a265e-d25e-41ae-82f5-4e1880a0ca29","Type":"ContainerDied","Data":"366006eba7b02e7b5b7829de0f2984582a721b1ddbfe6ba644e0b76967a420a8"} Oct 30 02:54:47.630336 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:54:47.630242 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:54:47.642480 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:47.642442 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e026bb-879a-4fc2-b0b6-5322d024de95" path="/var/lib/kubelet/pods/49e026bb-879a-4fc2-b0b6-5322d024de95/volumes" Oct 30 02:54:48.662687 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.662655 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:48.745609 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.745490 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-xks-ca\") pod \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " Oct 30 02:54:48.745609 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.745607 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-openshift-ca\") pod \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " Oct 30 02:54:48.745814 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.745765 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42a265e-d25e-41ae-82f5-4e1880a0ca29-kube-api-access-6tqwq\" (UniqueName: \"kubernetes.io/projected/e42a265e-d25e-41ae-82f5-4e1880a0ca29-kube-api-access-6tqwq\") pod \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\" (UID: \"e42a265e-d25e-41ae-82f5-4e1880a0ca29\") " Oct 30 02:54:48.746039 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.746009 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-openshift-ca" pod "e42a265e-d25e-41ae-82f5-4e1880a0ca29" (UID: "e42a265e-d25e-41ae-82f5-4e1880a0ca29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:54:48.746168 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.746153 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:48.747985 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.747948 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42a265e-d25e-41ae-82f5-4e1880a0ca29-kube-api-access-6tqwq" pod "e42a265e-d25e-41ae-82f5-4e1880a0ca29" (UID: "e42a265e-d25e-41ae-82f5-4e1880a0ca29"). InnerVolumeSpecName "kube-api-access-6tqwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:54:48.747985 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.747962 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-xks-ca" pod "e42a265e-d25e-41ae-82f5-4e1880a0ca29" (UID: "e42a265e-d25e-41ae-82f5-4e1880a0ca29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:54:48.847193 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.847146 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tqwq\" (UniqueName: \"kubernetes.io/projected/e42a265e-d25e-41ae-82f5-4e1880a0ca29-kube-api-access-6tqwq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:48.847193 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:48.847187 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42a265e-d25e-41ae-82f5-4e1880a0ca29-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:54:49.503985 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:49.503947 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" Oct 30 02:54:49.504194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:49.503977 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5" event={"ID":"e42a265e-d25e-41ae-82f5-4e1880a0ca29","Type":"ContainerDied","Data":"fc12fddf0af77275a502c1bb6520a18349b126294f780548a98292a165cd2cd2"} Oct 30 02:54:49.504194 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:54:49.504039 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc12fddf0af77275a502c1bb6520a18349b126294f780548a98292a165cd2cd2" Oct 30 02:55:11.048743 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:11.048642 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8"] Oct 30 02:55:11.060579 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:11.060504 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4f4c8"] Oct 30 02:55:11.641758 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:11.641719 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6737d10-6c9f-4388-846f-85ee2f96e21c" path="/var/lib/kubelet/pods/f6737d10-6c9f-4388-846f-85ee2f96e21c/volumes" Oct 30 02:55:12.979732 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:12.979688 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-56f8ccc945-54crd"] Oct 30 02:55:12.980180 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:12.980165 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e42a265e-d25e-41ae-82f5-4e1880a0ca29" containerName="revoke-keys" Oct 30 02:55:12.980225 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:12.980180 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42a265e-d25e-41ae-82f5-4e1880a0ca29" containerName="revoke-keys" Oct 30 02:55:12.980289 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:12.980278 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42a265e-d25e-41ae-82f5-4e1880a0ca29" containerName="revoke-keys" Oct 30 02:55:12.983873 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:12.983848 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:13.001073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.001000 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8725b087-ad98-4b8e-8c30-c3c62fb0b739-plugins-config-volume\") pod \"payload-processing-56f8ccc945-54crd\" (UID: \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\") " pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:13.001368 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.001102 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r79s\" (UniqueName: \"kubernetes.io/projected/8725b087-ad98-4b8e-8c30-c3c62fb0b739-kube-api-access-5r79s\") pod \"payload-processing-56f8ccc945-54crd\" (UID: \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\") " pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:13.001484 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.001426 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 02:55:13.001739 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.001708 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jrbb4\"" Oct 30 02:55:13.015791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.015754 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-56f8ccc945-54crd"] Oct 30 02:55:13.101838 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.101794 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r79s\" (UniqueName: \"kubernetes.io/projected/8725b087-ad98-4b8e-8c30-c3c62fb0b739-kube-api-access-5r79s\") pod \"payload-processing-56f8ccc945-54crd\" (UID: \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\") " pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:13.102063 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.101922 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8725b087-ad98-4b8e-8c30-c3c62fb0b739-plugins-config-volume\") pod \"payload-processing-56f8ccc945-54crd\" (UID: \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\") " pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:13.102651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.102631 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8725b087-ad98-4b8e-8c30-c3c62fb0b739-plugins-config-volume\") pod \"payload-processing-56f8ccc945-54crd\" (UID: \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\") " pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:13.125734 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.125706 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r79s\" (UniqueName: \"kubernetes.io/projected/8725b087-ad98-4b8e-8c30-c3c62fb0b739-kube-api-access-5r79s\") pod \"payload-processing-56f8ccc945-54crd\" (UID: \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\") " pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:13.297467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.297363 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:13.604320 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.604289 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-56f8ccc945-54crd"] Oct 30 02:55:13.607280 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:55:13.607243 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8725b087_ad98_4b8e_8c30_c3c62fb0b739.slice/crio-dbaf764a4c8b347eebda831a2d1d89e7106c9e1f0b6ddebef3764244c9028cce WatchSource:0}: Error finding container dbaf764a4c8b347eebda831a2d1d89e7106c9e1f0b6ddebef3764244c9028cce: Status 404 returned error can't find the container with id dbaf764a4c8b347eebda831a2d1d89e7106c9e1f0b6ddebef3764244c9028cce Oct 30 02:55:13.644420 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:13.644376 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" event={"ID":"8725b087-ad98-4b8e-8c30-c3c62fb0b739","Type":"ContainerStarted","Data":"dbaf764a4c8b347eebda831a2d1d89e7106c9e1f0b6ddebef3764244c9028cce"} Oct 30 02:55:14.651398 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:14.651352 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" event={"ID":"8725b087-ad98-4b8e-8c30-c3c62fb0b739","Type":"ContainerStarted","Data":"c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac"} Oct 30 02:55:14.651886 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:14.651460 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:14.694493 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:14.694429 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" podStartSLOduration=2.6944142319999997 podStartE2EDuration="2.694414232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:14.691753195 +0000 UTC m=+5193.194685415" watchObservedRunningTime="2026-10-30 02:55:14.694414232 +0000 UTC m=+5193.197346430" Oct 30 02:55:17.628110 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:55:17.628064 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:55:25.659796 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:25.659759 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:27.032896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.032851 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6"] Oct 30 02:55:27.044783 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.044745 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvsp6"] Oct 30 02:55:27.642221 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.642187 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdcde04b-22d9-41c0-9e82-434834ddc708" path="/var/lib/kubelet/pods/bdcde04b-22d9-41c0-9e82-434834ddc708/volumes" Oct 30 02:55:27.784849 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.784803 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl"] Oct 30 02:55:27.790700 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.790664 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:27.804485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.804451 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl"] Oct 30 02:55:27.850066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.850022 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6b84a83-7da1-4dfa-bb59-2ad03496cc4a-plugins-config-volume\") pod \"payload-pre-processing-7bf6df7cf-qcpgl\" (UID: \"a6b84a83-7da1-4dfa-bb59-2ad03496cc4a\") " pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:27.850066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.850074 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2km\" (UniqueName: \"kubernetes.io/projected/a6b84a83-7da1-4dfa-bb59-2ad03496cc4a-kube-api-access-2d2km\") pod \"payload-pre-processing-7bf6df7cf-qcpgl\" (UID: \"a6b84a83-7da1-4dfa-bb59-2ad03496cc4a\") " pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:27.950911 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.950876 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6b84a83-7da1-4dfa-bb59-2ad03496cc4a-plugins-config-volume\") pod \"payload-pre-processing-7bf6df7cf-qcpgl\" (UID: \"a6b84a83-7da1-4dfa-bb59-2ad03496cc4a\") " pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:27.951080 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.950922 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2km\" (UniqueName: \"kubernetes.io/projected/a6b84a83-7da1-4dfa-bb59-2ad03496cc4a-kube-api-access-2d2km\") pod \"payload-pre-processing-7bf6df7cf-qcpgl\" (UID: \"a6b84a83-7da1-4dfa-bb59-2ad03496cc4a\") " pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:27.951685 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.951663 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6b84a83-7da1-4dfa-bb59-2ad03496cc4a-plugins-config-volume\") pod \"payload-pre-processing-7bf6df7cf-qcpgl\" (UID: \"a6b84a83-7da1-4dfa-bb59-2ad03496cc4a\") " pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:27.974912 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:27.974875 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2km\" (UniqueName: \"kubernetes.io/projected/a6b84a83-7da1-4dfa-bb59-2ad03496cc4a-kube-api-access-2d2km\") pod \"payload-pre-processing-7bf6df7cf-qcpgl\" (UID: \"a6b84a83-7da1-4dfa-bb59-2ad03496cc4a\") " pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:28.105352 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:28.105309 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:28.414749 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:28.414714 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl"] Oct 30 02:55:28.417162 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:55:28.417130 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b84a83_7da1_4dfa_bb59_2ad03496cc4a.slice/crio-683dba2467befba1e4d7947ef005ff3f7c9ab7f29fe07d67b9774202b474ecb0 WatchSource:0}: Error finding container 683dba2467befba1e4d7947ef005ff3f7c9ab7f29fe07d67b9774202b474ecb0: Status 404 returned error can't find the container with id 683dba2467befba1e4d7947ef005ff3f7c9ab7f29fe07d67b9774202b474ecb0 Oct 30 02:55:28.740992 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:28.740952 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" event={"ID":"a6b84a83-7da1-4dfa-bb59-2ad03496cc4a","Type":"ContainerStarted","Data":"82a4fc4f1dc31e31b7d4726d0f77538375ab0a453b3d9b71e5039e422141158c"} Oct 30 02:55:28.741249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:28.741015 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" event={"ID":"a6b84a83-7da1-4dfa-bb59-2ad03496cc4a","Type":"ContainerStarted","Data":"683dba2467befba1e4d7947ef005ff3f7c9ab7f29fe07d67b9774202b474ecb0"} Oct 30 02:55:28.741249 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:28.741187 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:28.784498 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:28.784028 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" podStartSLOduration=1.7840055860000001 podStartE2EDuration="1.784005586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:28.782600134 +0000 UTC m=+5207.285532350" watchObservedRunningTime="2026-10-30 02:55:28.784005586 +0000 UTC m=+5207.286937806" Oct 30 02:55:38.032695 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:38.032655 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669"] Oct 30 02:55:38.044347 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:38.044305 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-mg669"] Oct 30 02:55:39.426192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:39.426146 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-56f8ccc945-54crd"] Oct 30 02:55:39.641880 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:39.641842 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3768308b-c017-4359-a598-a487135cf8fd" path="/var/lib/kubelet/pods/3768308b-c017-4359-a598-a487135cf8fd/volumes" Oct 30 02:55:39.750323 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:39.750290 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7bf6df7cf-qcpgl" Oct 30 02:55:44.426861 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.426811 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" podUID="8725b087-ad98-4b8e-8c30-c3c62fb0b739" containerName="payload-processing" containerID="cri-o://c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac" gracePeriod=25 Oct 30 02:55:44.702105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.702003 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:44.838366 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.838316 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8725b087-ad98-4b8e-8c30-c3c62fb0b739-kube-api-access-5r79s\" (UniqueName: \"kubernetes.io/projected/8725b087-ad98-4b8e-8c30-c3c62fb0b739-kube-api-access-5r79s\") pod \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\" (UID: \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\") " Oct 30 02:55:44.838636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.838413 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8725b087-ad98-4b8e-8c30-c3c62fb0b739-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8725b087-ad98-4b8e-8c30-c3c62fb0b739-plugins-config-volume\") pod \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\" (UID: \"8725b087-ad98-4b8e-8c30-c3c62fb0b739\") " Oct 30 02:55:44.839305 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.839263 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8725b087-ad98-4b8e-8c30-c3c62fb0b739-plugins-config-volume" pod "8725b087-ad98-4b8e-8c30-c3c62fb0b739" (UID: "8725b087-ad98-4b8e-8c30-c3c62fb0b739"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:55:44.841388 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.841351 5035 generic.go:356] "Generic (PLEG): container finished" podID="8725b087-ad98-4b8e-8c30-c3c62fb0b739" containerID="c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac" exitCode=0 Oct 30 02:55:44.841609 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.841585 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" event={"ID":"8725b087-ad98-4b8e-8c30-c3c62fb0b739","Type":"ContainerDied","Data":"c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac"} Oct 30 02:55:44.841709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.841653 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" event={"ID":"8725b087-ad98-4b8e-8c30-c3c62fb0b739","Type":"ContainerDied","Data":"dbaf764a4c8b347eebda831a2d1d89e7106c9e1f0b6ddebef3764244c9028cce"} Oct 30 02:55:44.841709 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.841687 5035 scope.go:122] "RemoveContainer" containerID="c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac" Oct 30 02:55:44.841888 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.841872 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-56f8ccc945-54crd" Oct 30 02:55:44.845139 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.845090 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8725b087-ad98-4b8e-8c30-c3c62fb0b739-kube-api-access-5r79s" pod "8725b087-ad98-4b8e-8c30-c3c62fb0b739" (UID: "8725b087-ad98-4b8e-8c30-c3c62fb0b739"). InnerVolumeSpecName "kube-api-access-5r79s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:55:44.865649 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.865599 5035 scope.go:122] "RemoveContainer" containerID="c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac" Oct 30 02:55:44.866244 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:55:44.866201 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac\": container with ID starting with c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac not found: ID does not exist" containerID="c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac" Oct 30 02:55:44.866378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.866261 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac"} err="failed to get container status \"c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac\": rpc error: code = NotFound desc = could not find container \"c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac\": container with ID starting with c7331b09a6b14ef5507c936972b0f8076da016f38bd53b172a6537c45d101dac not found: ID does not exist" Oct 30 02:55:44.940100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.940049 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r79s\" (UniqueName: \"kubernetes.io/projected/8725b087-ad98-4b8e-8c30-c3c62fb0b739-kube-api-access-5r79s\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:55:44.940100 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:44.940100 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8725b087-ad98-4b8e-8c30-c3c62fb0b739-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:55:45.160902 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:45.160852 5035 scope.go:122] "RemoveContainer" containerID="5e55c8c968b4bd3566222c126bb7c5c8d83d5d1b1c130ff6f34c0943c60c572e" Oct 30 02:55:45.177485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:45.177449 5035 scope.go:122] "RemoveContainer" containerID="9302c78bf97f89fc392628660f16f32f4dc36307c33e6bf7727aa39c1618f599" Oct 30 02:55:45.210936 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:45.210838 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-56f8ccc945-54crd"] Oct 30 02:55:45.214799 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:45.214765 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-56f8ccc945-54crd"] Oct 30 02:55:45.802552 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:55:45.802288 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8725b087-ad98-4b8e-8c30-c3c62fb0b739" path="/var/lib/kubelet/pods/8725b087-ad98-4b8e-8c30-c3c62fb0b739/volumes" Oct 30 02:55:47.652728 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:55:47.652688 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:56:17.630728 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:56:17.630686 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:56:41.032679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:56:41.032632 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb"] Oct 30 02:56:41.044341 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:56:41.044301 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0549523-8xvpb"] Oct 30 02:56:41.643688 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:56:41.643644 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff61e09-7a60-4cb4-b53a-8c7c9da66507" path="/var/lib/kubelet/pods/8ff61e09-7a60-4cb4-b53a-8c7c9da66507/volumes" Oct 30 02:56:45.254442 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:56:45.254406 5035 scope.go:122] "RemoveContainer" containerID="36d58554cbd6c9fde69da7467816c1b072c2e6a3a85dc98122390ab422b1c380" Oct 30 02:56:45.264007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:56:45.263980 5035 scope.go:122] "RemoveContainer" containerID="81aac3409df5dc35a9d75444a94efff54082c034887c7c4d2cbb1cef8763f40f" Oct 30 02:56:45.272987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:56:45.272964 5035 scope.go:122] "RemoveContainer" containerID="75896c11027950cab79464d54849a9df692fa9b9b51c9e602b13f2eb27757a82" Oct 30 02:56:47.630197 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:56:47.630152 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:56:57.031573 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:56:57.031518 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj"] Oct 30 02:56:57.043322 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:56:57.043286 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea923b9-7v5sj"] Oct 30 02:56:57.641643 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:56:57.641607 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b139cf64-e57c-4a19-a31c-8fc68e6155f7" path="/var/lib/kubelet/pods/b139cf64-e57c-4a19-a31c-8fc68e6155f7/volumes" Oct 30 02:57:17.629571 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:57:17.629502 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:57:23.267075 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.267035 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh"] Oct 30 02:57:23.267508 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.267494 5035 cpu_manager.go:414] "containerMap: removing container" podUID="8725b087-ad98-4b8e-8c30-c3c62fb0b739" containerName="payload-processing" Oct 30 02:57:23.267589 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.267509 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8725b087-ad98-4b8e-8c30-c3c62fb0b739" containerName="payload-processing" Oct 30 02:57:23.267645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.267634 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="8725b087-ad98-4b8e-8c30-c3c62fb0b739" containerName="payload-processing" Oct 30 02:57:23.271013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.270988 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.298116 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.298084 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:57:23.298370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.298111 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:57:23.298370 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.298148 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:57:23.298597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.298176 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:57:23.300686 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.300656 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh"] Oct 30 02:57:23.439411 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.439291 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-5trjh\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.445197 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.439431 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vtl\" (UniqueName: \"kubernetes.io/projected/e3d4574d-d573-46dd-bea0-c139be3fa2e3-kube-api-access-c4vtl\") pod \"maas-api-revoke-sub-e2547622-5trjh\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.445197 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.439465 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-5trjh\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.540710 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.540602 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-5trjh\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.540710 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.540708 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vtl\" (UniqueName: \"kubernetes.io/projected/e3d4574d-d573-46dd-bea0-c139be3fa2e3-kube-api-access-c4vtl\") pod \"maas-api-revoke-sub-e2547622-5trjh\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.540958 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.540742 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-5trjh\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.541397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.541373 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-5trjh\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.543278 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.543255 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-5trjh\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.564670 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.564637 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vtl\" (UniqueName: \"kubernetes.io/projected/e3d4574d-d573-46dd-bea0-c139be3fa2e3-kube-api-access-c4vtl\") pod \"maas-api-revoke-sub-e2547622-5trjh\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.584657 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.584621 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:23.893681 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:23.893650 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh"] Oct 30 02:57:23.895782 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:57:23.895749 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d4574d_d573_46dd_bea0_c139be3fa2e3.slice/crio-5f3bf9861409f8b3c3157198c941c765d641f979ca30ae7225a1cd67884fdb24 WatchSource:0}: Error finding container 5f3bf9861409f8b3c3157198c941c765d641f979ca30ae7225a1cd67884fdb24: Status 404 returned error can't find the container with id 5f3bf9861409f8b3c3157198c941c765d641f979ca30ae7225a1cd67884fdb24 Oct 30 02:57:24.450724 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:24.450688 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" event={"ID":"e3d4574d-d573-46dd-bea0-c139be3fa2e3","Type":"ContainerStarted","Data":"3da2f247a0091affbf7c2615138e2d47641406f85d0b67b01a0f4158271d0a42"} Oct 30 02:57:24.451163 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:24.450746 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" event={"ID":"e3d4574d-d573-46dd-bea0-c139be3fa2e3","Type":"ContainerStarted","Data":"5f3bf9861409f8b3c3157198c941c765d641f979ca30ae7225a1cd67884fdb24"} Oct 30 02:57:24.493537 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:24.493474 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" podStartSLOduration=1.493457423 podStartE2EDuration="1.493457423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:24.491195792 +0000 UTC m=+5322.994128029" watchObservedRunningTime="2026-10-30 02:57:24.493457423 +0000 UTC m=+5322.996389642" Oct 30 02:57:25.457401 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:25.457362 5035 generic.go:356] "Generic (PLEG): container finished" podID="e3d4574d-d573-46dd-bea0-c139be3fa2e3" containerID="3da2f247a0091affbf7c2615138e2d47641406f85d0b67b01a0f4158271d0a42" exitCode=0 Oct 30 02:57:25.457881 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:25.457446 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" event={"ID":"e3d4574d-d573-46dd-bea0-c139be3fa2e3","Type":"ContainerDied","Data":"3da2f247a0091affbf7c2615138e2d47641406f85d0b67b01a0f4158271d0a42"} Oct 30 02:57:26.616817 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.616787 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:26.775789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.775691 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-openshift-ca\") pod \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " Oct 30 02:57:26.775789 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.775740 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-xks-ca\") pod \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " Oct 30 02:57:26.776012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.775800 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d4574d-d573-46dd-bea0-c139be3fa2e3-kube-api-access-c4vtl\" (UniqueName: \"kubernetes.io/projected/e3d4574d-d573-46dd-bea0-c139be3fa2e3-kube-api-access-c4vtl\") pod \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\" (UID: \"e3d4574d-d573-46dd-bea0-c139be3fa2e3\") " Oct 30 02:57:26.776178 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.776146 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-openshift-ca" pod "e3d4574d-d573-46dd-bea0-c139be3fa2e3" (UID: "e3d4574d-d573-46dd-bea0-c139be3fa2e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:26.778042 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.778021 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-xks-ca" pod "e3d4574d-d573-46dd-bea0-c139be3fa2e3" (UID: "e3d4574d-d573-46dd-bea0-c139be3fa2e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:26.778128 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.778099 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d4574d-d573-46dd-bea0-c139be3fa2e3-kube-api-access-c4vtl" pod "e3d4574d-d573-46dd-bea0-c139be3fa2e3" (UID: "e3d4574d-d573-46dd-bea0-c139be3fa2e3"). InnerVolumeSpecName "kube-api-access-c4vtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:26.877011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.876961 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:57:26.877011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.877003 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d4574d-d573-46dd-bea0-c139be3fa2e3-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:57:26.877011 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:26.877021 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vtl\" (UniqueName: \"kubernetes.io/projected/e3d4574d-d573-46dd-bea0-c139be3fa2e3-kube-api-access-c4vtl\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:57:27.469839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:27.469805 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" Oct 30 02:57:27.469839 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:27.469833 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh" event={"ID":"e3d4574d-d573-46dd-bea0-c139be3fa2e3","Type":"ContainerDied","Data":"5f3bf9861409f8b3c3157198c941c765d641f979ca30ae7225a1cd67884fdb24"} Oct 30 02:57:27.470119 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:27.469888 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f3bf9861409f8b3c3157198c941c765d641f979ca30ae7225a1cd67884fdb24" Oct 30 02:57:45.307566 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:45.307499 5035 scope.go:122] "RemoveContainer" containerID="ed7ef7dbf0a3ee361536c608c00134f38f0811b1cf9414f11ede5e9793e93803" Oct 30 02:57:46.803007 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.802954 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms"] Oct 30 02:57:46.803701 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.803675 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e3d4574d-d573-46dd-bea0-c139be3fa2e3" containerName="revoke-keys" Oct 30 02:57:46.803807 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.803702 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d4574d-d573-46dd-bea0-c139be3fa2e3" containerName="revoke-keys" Oct 30 02:57:46.803869 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.803846 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d4574d-d573-46dd-bea0-c139be3fa2e3" containerName="revoke-keys" Oct 30 02:57:46.808797 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.808768 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:46.824562 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.824449 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:57:46.836684 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.836653 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:57:46.836918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.836651 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:57:46.837008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.836747 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:57:46.840827 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.840788 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms"] Oct 30 02:57:46.919813 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.919769 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-665ms\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:46.919813 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.919813 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdkr\" (UniqueName: \"kubernetes.io/projected/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-kube-api-access-sfdkr\") pod \"maas-api-revoke-sub-c1e271a9-665ms\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:46.920118 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:46.919990 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-665ms\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:47.021185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:47.021140 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-665ms\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:47.021185 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:47.021187 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdkr\" (UniqueName: \"kubernetes.io/projected/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-kube-api-access-sfdkr\") pod \"maas-api-revoke-sub-c1e271a9-665ms\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:47.021432 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:47.021243 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-665ms\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:47.021962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:47.021930 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-665ms\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:47.023644 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:47.023614 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-665ms\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:47.045094 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:47.045060 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdkr\" (UniqueName: \"kubernetes.io/projected/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-kube-api-access-sfdkr\") pod \"maas-api-revoke-sub-c1e271a9-665ms\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:47.123000 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:47.122877 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:47.429834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:47.429799 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms"] Oct 30 02:57:47.431794 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:57:47.431758 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2cafdd_d7e8_4475_a5d7_58c4b1006fdb.slice/crio-07d0a445e5141476088c3e44057d120b1b80adf00fffd0314f875dfbdb371df4 WatchSource:0}: Error finding container 07d0a445e5141476088c3e44057d120b1b80adf00fffd0314f875dfbdb371df4: Status 404 returned error can't find the container with id 07d0a445e5141476088c3e44057d120b1b80adf00fffd0314f875dfbdb371df4 Oct 30 02:57:47.629718 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:57:47.629678 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:57:47.643310 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:47.643272 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" event={"ID":"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb","Type":"ContainerStarted","Data":"07d0a445e5141476088c3e44057d120b1b80adf00fffd0314f875dfbdb371df4"} Oct 30 02:57:48.650464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:48.650419 5035 generic.go:356] "Generic (PLEG): container finished" podID="cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb" containerID="914bcafcba8a125e37ba021fb3a360bcd4a7559e63630a4d56a1b72971122035" exitCode=0 Oct 30 02:57:48.650967 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:48.650508 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" event={"ID":"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb","Type":"ContainerDied","Data":"914bcafcba8a125e37ba021fb3a360bcd4a7559e63630a4d56a1b72971122035"} Oct 30 02:57:49.807901 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:49.807865 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:49.951972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:49.951920 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-kube-api-access-sfdkr\" (UniqueName: \"kubernetes.io/projected/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-kube-api-access-sfdkr\") pod \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " Oct 30 02:57:49.952155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:49.952052 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-openshift-ca\") pod \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " Oct 30 02:57:49.952155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:49.952084 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-xks-ca\") pod \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\" (UID: \"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb\") " Oct 30 02:57:49.952482 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:49.952451 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-openshift-ca" pod "cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb" (UID: "cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:49.954308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:49.954279 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-xks-ca" pod "cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb" (UID: "cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:49.954439 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:49.954285 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-kube-api-access-sfdkr" pod "cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb" (UID: "cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb"). InnerVolumeSpecName "kube-api-access-sfdkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:50.053925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:50.053883 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:57:50.053925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:50.053920 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:57:50.053925 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:50.053929 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfdkr\" (UniqueName: \"kubernetes.io/projected/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb-kube-api-access-sfdkr\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:57:50.663996 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:50.663957 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" Oct 30 02:57:50.664203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:50.663971 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms" event={"ID":"cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb","Type":"ContainerDied","Data":"07d0a445e5141476088c3e44057d120b1b80adf00fffd0314f875dfbdb371df4"} Oct 30 02:57:50.664203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:57:50.664083 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07d0a445e5141476088c3e44057d120b1b80adf00fffd0314f875dfbdb371df4" Oct 30 02:58:06.685097 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.685055 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr"] Oct 30 02:58:06.685677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.685633 5035 cpu_manager.go:414] "containerMap: removing container" podUID="cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb" containerName="revoke-keys" Oct 30 02:58:06.685677 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.685652 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb" containerName="revoke-keys" Oct 30 02:58:06.685778 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.685722 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb" containerName="revoke-keys" Oct 30 02:58:06.689184 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.689158 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:06.716372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.716291 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:58:06.716372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.716344 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:58:06.716372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.716322 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:58:06.716372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.716307 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:58:06.719962 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.719895 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr"] Oct 30 02:58:06.804918 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.804880 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5rd\" (UniqueName: \"kubernetes.io/projected/710325c1-2556-4e53-b0be-e71662734b17-kube-api-access-dw5rd\") pod \"maas-api-revoke-sub-2e1f5b5f-2crjr\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:06.805148 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.804951 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710325c1-2556-4e53-b0be-e71662734b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2crjr\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:06.805148 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.805070 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710325c1-2556-4e53-b0be-e71662734b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2crjr\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:06.905739 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.905700 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710325c1-2556-4e53-b0be-e71662734b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2crjr\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:06.905976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.905790 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5rd\" (UniqueName: \"kubernetes.io/projected/710325c1-2556-4e53-b0be-e71662734b17-kube-api-access-dw5rd\") pod \"maas-api-revoke-sub-2e1f5b5f-2crjr\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:06.905976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.905847 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710325c1-2556-4e53-b0be-e71662734b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2crjr\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:06.906492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.906460 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710325c1-2556-4e53-b0be-e71662734b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2crjr\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:06.908375 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.908350 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710325c1-2556-4e53-b0be-e71662734b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2crjr\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:06.929658 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:06.929620 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5rd\" (UniqueName: \"kubernetes.io/projected/710325c1-2556-4e53-b0be-e71662734b17-kube-api-access-dw5rd\") pod \"maas-api-revoke-sub-2e1f5b5f-2crjr\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:07.003321 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:07.003280 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:07.310755 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:07.310722 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr"] Oct 30 02:58:07.313361 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:58:07.313317 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710325c1_2556_4e53_b0be_e71662734b17.slice/crio-cd5feb4702409128952df532b6b63a9baa23ac055363f95ba7c65f564d36f682 WatchSource:0}: Error finding container cd5feb4702409128952df532b6b63a9baa23ac055363f95ba7c65f564d36f682: Status 404 returned error can't find the container with id cd5feb4702409128952df532b6b63a9baa23ac055363f95ba7c65f564d36f682 Oct 30 02:58:07.810291 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:07.810252 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" event={"ID":"710325c1-2556-4e53-b0be-e71662734b17","Type":"ContainerStarted","Data":"f11cfb957eb8506985b7fb5f988ee9ba438cee3e95721ef332b1e81ec31bf6f0"} Oct 30 02:58:07.810744 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:07.810315 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" event={"ID":"710325c1-2556-4e53-b0be-e71662734b17","Type":"ContainerStarted","Data":"cd5feb4702409128952df532b6b63a9baa23ac055363f95ba7c65f564d36f682"} Oct 30 02:58:07.852720 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:07.852602 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" podStartSLOduration=1.8525856539999999 podStartE2EDuration="1.852585654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:07.852509057 +0000 UTC m=+5366.355441339" watchObservedRunningTime="2026-10-30 02:58:07.852585654 +0000 UTC m=+5366.355517874" Oct 30 02:58:08.817923 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:08.817883 5035 generic.go:356] "Generic (PLEG): container finished" podID="710325c1-2556-4e53-b0be-e71662734b17" containerID="f11cfb957eb8506985b7fb5f988ee9ba438cee3e95721ef332b1e81ec31bf6f0" exitCode=0 Oct 30 02:58:08.818373 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:08.817941 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" event={"ID":"710325c1-2556-4e53-b0be-e71662734b17","Type":"ContainerDied","Data":"f11cfb957eb8506985b7fb5f988ee9ba438cee3e95721ef332b1e81ec31bf6f0"} Oct 30 02:58:09.978805 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:09.978777 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:10.038222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.038184 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710325c1-2556-4e53-b0be-e71662734b17-kube-api-access-dw5rd\" (UniqueName: \"kubernetes.io/projected/710325c1-2556-4e53-b0be-e71662734b17-kube-api-access-dw5rd\") pod \"710325c1-2556-4e53-b0be-e71662734b17\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " Oct 30 02:58:10.038458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.038265 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/710325c1-2556-4e53-b0be-e71662734b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710325c1-2556-4e53-b0be-e71662734b17-maas-api-xks-ca\") pod \"710325c1-2556-4e53-b0be-e71662734b17\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " Oct 30 02:58:10.038458 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.038366 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710325c1-2556-4e53-b0be-e71662734b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710325c1-2556-4e53-b0be-e71662734b17-maas-api-openshift-ca\") pod \"710325c1-2556-4e53-b0be-e71662734b17\" (UID: \"710325c1-2556-4e53-b0be-e71662734b17\") " Oct 30 02:58:10.038777 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.038743 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710325c1-2556-4e53-b0be-e71662734b17-maas-api-openshift-ca" pod "710325c1-2556-4e53-b0be-e71662734b17" (UID: "710325c1-2556-4e53-b0be-e71662734b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:58:10.040506 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.040468 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/710325c1-2556-4e53-b0be-e71662734b17-maas-api-xks-ca" pod "710325c1-2556-4e53-b0be-e71662734b17" (UID: "710325c1-2556-4e53-b0be-e71662734b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:58:10.040665 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.040571 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710325c1-2556-4e53-b0be-e71662734b17-kube-api-access-dw5rd" pod "710325c1-2556-4e53-b0be-e71662734b17" (UID: "710325c1-2556-4e53-b0be-e71662734b17"). InnerVolumeSpecName "kube-api-access-dw5rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:58:10.140273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.140176 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw5rd\" (UniqueName: \"kubernetes.io/projected/710325c1-2556-4e53-b0be-e71662734b17-kube-api-access-dw5rd\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.140273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.140213 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710325c1-2556-4e53-b0be-e71662734b17-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.140273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.140226 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710325c1-2556-4e53-b0be-e71662734b17-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.830239 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.830207 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" Oct 30 02:58:10.830441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.830239 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr" event={"ID":"710325c1-2556-4e53-b0be-e71662734b17","Type":"ContainerDied","Data":"cd5feb4702409128952df532b6b63a9baa23ac055363f95ba7c65f564d36f682"} Oct 30 02:58:10.830441 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:10.830292 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd5feb4702409128952df532b6b63a9baa23ac055363f95ba7c65f564d36f682" Oct 30 02:58:17.627789 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:58:17.627742 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:58:17.940597 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.940479 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv"] Oct 30 02:58:17.941013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.940994 5035 cpu_manager.go:414] "containerMap: removing container" podUID="710325c1-2556-4e53-b0be-e71662734b17" containerName="revoke-keys" Oct 30 02:58:17.941013 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.941011 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710325c1-2556-4e53-b0be-e71662734b17" containerName="revoke-keys" Oct 30 02:58:17.941176 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.941108 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="710325c1-2556-4e53-b0be-e71662734b17" containerName="revoke-keys" Oct 30 02:58:17.944728 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.944702 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:17.961694 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.961655 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:58:17.961909 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.961760 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:58:17.981006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.980961 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv"] Oct 30 02:58:17.981492 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.981453 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:58:17.981935 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:17.981913 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:58:18.020587 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.020547 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828xd\" (UniqueName: \"kubernetes.io/projected/bf3622d2-10ba-48e8-8447-ffba317ab42e-kube-api-access-828xd\") pod \"maas-api-revoke-sub-ec843018-mfbdv\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.020791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.020610 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mfbdv\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.020791 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.020702 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mfbdv\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.122019 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.121971 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828xd\" (UniqueName: \"kubernetes.io/projected/bf3622d2-10ba-48e8-8447-ffba317ab42e-kube-api-access-828xd\") pod \"maas-api-revoke-sub-ec843018-mfbdv\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.122222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.122046 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mfbdv\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.122222 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.122090 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mfbdv\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.122819 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.122793 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mfbdv\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.124691 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.124664 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mfbdv\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.145260 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.145220 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828xd\" (UniqueName: \"kubernetes.io/projected/bf3622d2-10ba-48e8-8447-ffba317ab42e-kube-api-access-828xd\") pod \"maas-api-revoke-sub-ec843018-mfbdv\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.260696 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.260658 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:18.566810 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.566775 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv"] Oct 30 02:58:18.568689 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:58:18.568655 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3622d2_10ba_48e8_8447_ffba317ab42e.slice/crio-3a6184dbdeedad39b924f440a6fa8adf7417025e0d7264bfc119900b0232773d WatchSource:0}: Error finding container 3a6184dbdeedad39b924f440a6fa8adf7417025e0d7264bfc119900b0232773d: Status 404 returned error can't find the container with id 3a6184dbdeedad39b924f440a6fa8adf7417025e0d7264bfc119900b0232773d Oct 30 02:58:18.878254 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:18.878220 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" event={"ID":"bf3622d2-10ba-48e8-8447-ffba317ab42e","Type":"ContainerStarted","Data":"3a6184dbdeedad39b924f440a6fa8adf7417025e0d7264bfc119900b0232773d"} Oct 30 02:58:19.885974 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:19.885873 5035 generic.go:356] "Generic (PLEG): container finished" podID="bf3622d2-10ba-48e8-8447-ffba317ab42e" containerID="e03a79a0ad8933477672f560987edd3fb2ee5f43ab1eaee81880d6ac57b59fb1" exitCode=0 Oct 30 02:58:19.885974 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:19.885959 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" event={"ID":"bf3622d2-10ba-48e8-8447-ffba317ab42e","Type":"ContainerDied","Data":"e03a79a0ad8933477672f560987edd3fb2ee5f43ab1eaee81880d6ac57b59fb1"} Oct 30 02:58:21.054624 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.054594 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:21.157712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.157607 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-openshift-ca\") pod \"bf3622d2-10ba-48e8-8447-ffba317ab42e\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " Oct 30 02:58:21.157712 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.157704 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-xks-ca\") pod \"bf3622d2-10ba-48e8-8447-ffba317ab42e\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " Oct 30 02:58:21.157987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.157794 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3622d2-10ba-48e8-8447-ffba317ab42e-kube-api-access-828xd\" (UniqueName: \"kubernetes.io/projected/bf3622d2-10ba-48e8-8447-ffba317ab42e-kube-api-access-828xd\") pod \"bf3622d2-10ba-48e8-8447-ffba317ab42e\" (UID: \"bf3622d2-10ba-48e8-8447-ffba317ab42e\") " Oct 30 02:58:21.158058 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.157994 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-openshift-ca" pod "bf3622d2-10ba-48e8-8447-ffba317ab42e" (UID: "bf3622d2-10ba-48e8-8447-ffba317ab42e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:58:21.158227 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.158209 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:58:21.159973 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.159944 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-xks-ca" pod "bf3622d2-10ba-48e8-8447-ffba317ab42e" (UID: "bf3622d2-10ba-48e8-8447-ffba317ab42e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:58:21.160094 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.159992 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3622d2-10ba-48e8-8447-ffba317ab42e-kube-api-access-828xd" pod "bf3622d2-10ba-48e8-8447-ffba317ab42e" (UID: "bf3622d2-10ba-48e8-8447-ffba317ab42e"). InnerVolumeSpecName "kube-api-access-828xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:58:21.259031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.258989 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3622d2-10ba-48e8-8447-ffba317ab42e-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:58:21.259031 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.259026 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-828xd\" (UniqueName: \"kubernetes.io/projected/bf3622d2-10ba-48e8-8447-ffba317ab42e-kube-api-access-828xd\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:58:21.906940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.906838 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" event={"ID":"bf3622d2-10ba-48e8-8447-ffba317ab42e","Type":"ContainerDied","Data":"3a6184dbdeedad39b924f440a6fa8adf7417025e0d7264bfc119900b0232773d"} Oct 30 02:58:21.906940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.906900 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6184dbdeedad39b924f440a6fa8adf7417025e0d7264bfc119900b0232773d" Oct 30 02:58:21.906940 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:21.906857 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv" Oct 30 02:58:44.227388 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:44.227359 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:58:44.237895 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:44.237861 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:58:44.242217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:44.242191 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:58:44.339447 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:44.339418 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 02:58:44.350090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:44.350065 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 02:58:44.364904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:44.364868 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 02:58:45.400956 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.400911 5035 scope.go:122] "RemoveContainer" containerID="7a67cd7e45acabbef7fb36524d2c2d00ef499c044e2ce5e168bf1a8db48dcf4d" Oct 30 02:58:45.777878 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.777840 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t"] Oct 30 02:58:45.778433 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.778415 5035 cpu_manager.go:414] "containerMap: removing container" podUID="bf3622d2-10ba-48e8-8447-ffba317ab42e" containerName="revoke-keys" Oct 30 02:58:45.778490 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.778435 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3622d2-10ba-48e8-8447-ffba317ab42e" containerName="revoke-keys" Oct 30 02:58:45.778576 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.778564 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3622d2-10ba-48e8-8447-ffba317ab42e" containerName="revoke-keys" Oct 30 02:58:45.783581 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.783521 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:45.811109 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.811070 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:58:45.811310 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.811137 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:58:45.811310 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.811176 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:58:45.811310 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.811268 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:58:45.813989 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.813958 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t"] Oct 30 02:58:45.889670 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.889632 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4xn5t\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:45.889863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.889684 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9799x\" (UniqueName: \"kubernetes.io/projected/ebfbff74-8164-48c3-9dcb-76b9f3198923-kube-api-access-9799x\") pod \"maas-api-revoke-sub-046e1343-4xn5t\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:45.889863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.889796 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4xn5t\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:45.990751 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.990711 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4xn5t\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:45.990943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.990784 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9799x\" (UniqueName: \"kubernetes.io/projected/ebfbff74-8164-48c3-9dcb-76b9f3198923-kube-api-access-9799x\") pod \"maas-api-revoke-sub-046e1343-4xn5t\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:45.990943 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.990819 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4xn5t\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:45.991594 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.991561 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4xn5t\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:45.993368 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:45.993343 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4xn5t\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:46.016926 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:46.016888 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9799x\" (UniqueName: \"kubernetes.io/projected/ebfbff74-8164-48c3-9dcb-76b9f3198923-kube-api-access-9799x\") pod \"maas-api-revoke-sub-046e1343-4xn5t\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:46.097273 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:46.097169 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:46.404414 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:46.404380 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t"] Oct 30 02:58:46.406723 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:58:46.406687 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfbff74_8164_48c3_9dcb_76b9f3198923.slice/crio-2b634bcd67bcf43cf71a4b0638d91cc73ba8acba298fa4878f892a056da76b19 WatchSource:0}: Error finding container 2b634bcd67bcf43cf71a4b0638d91cc73ba8acba298fa4878f892a056da76b19: Status 404 returned error can't find the container with id 2b634bcd67bcf43cf71a4b0638d91cc73ba8acba298fa4878f892a056da76b19 Oct 30 02:58:47.094860 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:47.094816 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" event={"ID":"ebfbff74-8164-48c3-9dcb-76b9f3198923","Type":"ContainerStarted","Data":"be7793f5ddad31f158ec8a524a5d2b2f38b031f6b8d5545f416af03565c47add"} Oct 30 02:58:47.095061 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:47.094880 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" event={"ID":"ebfbff74-8164-48c3-9dcb-76b9f3198923","Type":"ContainerStarted","Data":"2b634bcd67bcf43cf71a4b0638d91cc73ba8acba298fa4878f892a056da76b19"} Oct 30 02:58:47.144700 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:47.144627 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" podStartSLOduration=2.14460601 podStartE2EDuration="2.14460601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:58:47.143071506 +0000 UTC m=+5405.646003725" watchObservedRunningTime="2026-10-30 02:58:47.14460601 +0000 UTC m=+5405.647538230" Oct 30 02:58:47.631066 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:58:47.631020 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:58:48.102210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:48.102170 5035 generic.go:356] "Generic (PLEG): container finished" podID="ebfbff74-8164-48c3-9dcb-76b9f3198923" containerID="be7793f5ddad31f158ec8a524a5d2b2f38b031f6b8d5545f416af03565c47add" exitCode=0 Oct 30 02:58:48.102422 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:48.102251 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" event={"ID":"ebfbff74-8164-48c3-9dcb-76b9f3198923","Type":"ContainerDied","Data":"be7793f5ddad31f158ec8a524a5d2b2f38b031f6b8d5545f416af03565c47add"} Oct 30 02:58:49.259115 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.259085 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:49.430022 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.429921 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-xks-ca\") pod \"ebfbff74-8164-48c3-9dcb-76b9f3198923\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " Oct 30 02:58:49.430192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.430033 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-openshift-ca\") pod \"ebfbff74-8164-48c3-9dcb-76b9f3198923\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " Oct 30 02:58:49.430192 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.430066 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebfbff74-8164-48c3-9dcb-76b9f3198923-kube-api-access-9799x\" (UniqueName: \"kubernetes.io/projected/ebfbff74-8164-48c3-9dcb-76b9f3198923-kube-api-access-9799x\") pod \"ebfbff74-8164-48c3-9dcb-76b9f3198923\" (UID: \"ebfbff74-8164-48c3-9dcb-76b9f3198923\") " Oct 30 02:58:49.430409 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.430379 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-openshift-ca" pod "ebfbff74-8164-48c3-9dcb-76b9f3198923" (UID: "ebfbff74-8164-48c3-9dcb-76b9f3198923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:58:49.432092 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.432067 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-xks-ca" pod "ebfbff74-8164-48c3-9dcb-76b9f3198923" (UID: "ebfbff74-8164-48c3-9dcb-76b9f3198923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:58:49.432203 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.432176 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfbff74-8164-48c3-9dcb-76b9f3198923-kube-api-access-9799x" pod "ebfbff74-8164-48c3-9dcb-76b9f3198923" (UID: "ebfbff74-8164-48c3-9dcb-76b9f3198923"). InnerVolumeSpecName "kube-api-access-9799x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:58:49.531460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.531414 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:58:49.531460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.531452 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebfbff74-8164-48c3-9dcb-76b9f3198923-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:58:49.531460 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:49.531465 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9799x\" (UniqueName: \"kubernetes.io/projected/ebfbff74-8164-48c3-9dcb-76b9f3198923-kube-api-access-9799x\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:58:50.116393 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:50.116360 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" Oct 30 02:58:50.116636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:50.116405 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t" event={"ID":"ebfbff74-8164-48c3-9dcb-76b9f3198923","Type":"ContainerDied","Data":"2b634bcd67bcf43cf71a4b0638d91cc73ba8acba298fa4878f892a056da76b19"} Oct 30 02:58:50.116636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:50.116488 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b634bcd67bcf43cf71a4b0638d91cc73ba8acba298fa4878f892a056da76b19" Oct 30 02:58:56.211877 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.211829 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt"] Oct 30 02:58:56.212583 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.212561 5035 cpu_manager.go:414] "containerMap: removing container" podUID="ebfbff74-8164-48c3-9dcb-76b9f3198923" containerName="revoke-keys" Oct 30 02:58:56.212651 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.212588 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebfbff74-8164-48c3-9dcb-76b9f3198923" containerName="revoke-keys" Oct 30 02:58:56.212781 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.212769 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebfbff74-8164-48c3-9dcb-76b9f3198923" containerName="revoke-keys" Oct 30 02:58:56.216739 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.216719 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.236961 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.236923 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt"] Oct 30 02:58:56.243250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.243184 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:58:56.243250 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.243207 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:58:56.307679 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.307637 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-62cwt\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.307876 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.307721 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-62cwt\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.307876 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.307825 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l27vn\" (UniqueName: \"kubernetes.io/projected/5d839b5d-c221-4e6a-9bd4-8bda77c68431-kube-api-access-l27vn\") pod \"maas-api-revoke-sub-d3fba056-62cwt\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.316831 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.316798 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:58:56.316976 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.316833 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:58:56.408656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.408611 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-62cwt\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.408656 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.408658 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-62cwt\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.408896 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.408687 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27vn\" (UniqueName: \"kubernetes.io/projected/5d839b5d-c221-4e6a-9bd4-8bda77c68431-kube-api-access-l27vn\") pod \"maas-api-revoke-sub-d3fba056-62cwt\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.409324 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.409297 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-62cwt\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.411211 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.411184 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-62cwt\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.432130 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.432091 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27vn\" (UniqueName: \"kubernetes.io/projected/5d839b5d-c221-4e6a-9bd4-8bda77c68431-kube-api-access-l27vn\") pod \"maas-api-revoke-sub-d3fba056-62cwt\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.535090 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.534994 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:58:56.841356 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:56.841322 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt"] Oct 30 02:58:56.843158 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:58:56.843126 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d839b5d_c221_4e6a_9bd4_8bda77c68431.slice/crio-522945b7a6ec2fe3f20e04eea0156300a71255582e6448607d441b54c57dc7ad WatchSource:0}: Error finding container 522945b7a6ec2fe3f20e04eea0156300a71255582e6448607d441b54c57dc7ad: Status 404 returned error can't find the container with id 522945b7a6ec2fe3f20e04eea0156300a71255582e6448607d441b54c57dc7ad Oct 30 02:58:57.158210 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:57.158112 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" event={"ID":"5d839b5d-c221-4e6a-9bd4-8bda77c68431","Type":"ContainerStarted","Data":"522945b7a6ec2fe3f20e04eea0156300a71255582e6448607d441b54c57dc7ad"} Oct 30 02:58:58.172570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:58.172432 5035 generic.go:356] "Generic (PLEG): container finished" podID="5d839b5d-c221-4e6a-9bd4-8bda77c68431" containerID="27f83057ae0b9be986dad7005a2ea8a52a872e2a005823eba1a6feb4783b1947" exitCode=0 Oct 30 02:58:58.172570 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:58:58.172522 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" event={"ID":"5d839b5d-c221-4e6a-9bd4-8bda77c68431","Type":"ContainerDied","Data":"27f83057ae0b9be986dad7005a2ea8a52a872e2a005823eba1a6feb4783b1947"} Oct 30 02:59:00.242461 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.242413 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" event={"ID":"5d839b5d-c221-4e6a-9bd4-8bda77c68431","Type":"ContainerDied","Data":"522945b7a6ec2fe3f20e04eea0156300a71255582e6448607d441b54c57dc7ad"} Oct 30 02:59:00.243717 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.243691 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="522945b7a6ec2fe3f20e04eea0156300a71255582e6448607d441b54c57dc7ad" Oct 30 02:59:00.243947 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.242498 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:59:00.272008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.271963 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d839b5d-c221-4e6a-9bd4-8bda77c68431-kube-api-access-l27vn\" (UniqueName: \"kubernetes.io/projected/5d839b5d-c221-4e6a-9bd4-8bda77c68431-kube-api-access-l27vn\") pod \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " Oct 30 02:59:00.272328 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.272302 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-xks-ca\") pod \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " Oct 30 02:59:00.272545 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.272504 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-openshift-ca\") pod \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\" (UID: \"5d839b5d-c221-4e6a-9bd4-8bda77c68431\") " Oct 30 02:59:00.273191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.273058 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-openshift-ca" pod "5d839b5d-c221-4e6a-9bd4-8bda77c68431" (UID: "5d839b5d-c221-4e6a-9bd4-8bda77c68431"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:59:00.273702 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.273663 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:00.283191 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.283143 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-xks-ca" pod "5d839b5d-c221-4e6a-9bd4-8bda77c68431" (UID: "5d839b5d-c221-4e6a-9bd4-8bda77c68431"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:59:00.283979 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.283890 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d839b5d-c221-4e6a-9bd4-8bda77c68431-kube-api-access-l27vn" pod "5d839b5d-c221-4e6a-9bd4-8bda77c68431" (UID: "5d839b5d-c221-4e6a-9bd4-8bda77c68431"). InnerVolumeSpecName "kube-api-access-l27vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:59:00.375202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.375153 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l27vn\" (UniqueName: \"kubernetes.io/projected/5d839b5d-c221-4e6a-9bd4-8bda77c68431-kube-api-access-l27vn\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:00.375202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:00.375195 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d839b5d-c221-4e6a-9bd4-8bda77c68431-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:01.248473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:01.248403 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt" Oct 30 02:59:10.032373 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:10.032330 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b"] Oct 30 02:59:10.044251 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:10.044206 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-szm8b"] Oct 30 02:59:11.642936 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:11.642895 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e2a7029-0880-48c1-8849-d744364ccc19" path="/var/lib/kubelet/pods/0e2a7029-0880-48c1-8849-d744364ccc19/volumes" Oct 30 02:59:17.660313 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:59:17.660216 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:59:17.838464 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.838413 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr"] Oct 30 02:59:17.838994 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.838971 5035 cpu_manager.go:414] "containerMap: removing container" podUID="5d839b5d-c221-4e6a-9bd4-8bda77c68431" containerName="revoke-keys" Oct 30 02:59:17.839105 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.838995 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d839b5d-c221-4e6a-9bd4-8bda77c68431" containerName="revoke-keys" Oct 30 02:59:17.839172 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.839155 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d839b5d-c221-4e6a-9bd4-8bda77c68431" containerName="revoke-keys" Oct 30 02:59:17.842806 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.842782 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:17.869735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.869696 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:59:17.870025 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.869717 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:59:17.870025 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.869778 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:59:17.870214 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.869782 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:59:17.872564 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.872520 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr"] Oct 30 02:59:17.936245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.936140 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzrj\" (UniqueName: \"kubernetes.io/projected/b2f0a2df-aef0-4a8a-8465-959dc18036c8-kube-api-access-gfzrj\") pod \"maas-api-revoke-sub-ae594979-qqxcr\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:17.936245 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.936194 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-qqxcr\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:17.936467 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:17.936261 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-qqxcr\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:18.037485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:18.037438 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzrj\" (UniqueName: \"kubernetes.io/projected/b2f0a2df-aef0-4a8a-8465-959dc18036c8-kube-api-access-gfzrj\") pod \"maas-api-revoke-sub-ae594979-qqxcr\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:18.037485 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:18.037486 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-qqxcr\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:18.037808 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:18.037602 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-qqxcr\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:18.038165 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:18.038142 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-qqxcr\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:18.040029 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:18.040002 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-qqxcr\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:18.061228 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:18.061191 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzrj\" (UniqueName: \"kubernetes.io/projected/b2f0a2df-aef0-4a8a-8465-959dc18036c8-kube-api-access-gfzrj\") pod \"maas-api-revoke-sub-ae594979-qqxcr\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:18.156469 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:18.156424 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:18.466642 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:18.466607 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr"] Oct 30 02:59:18.468814 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:59:18.468778 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f0a2df_aef0_4a8a_8465_959dc18036c8.slice/crio-7155889f52e4d135a6df02962d2ba6c3820541b5907b4ac972d5f3eee670a4c5 WatchSource:0}: Error finding container 7155889f52e4d135a6df02962d2ba6c3820541b5907b4ac972d5f3eee670a4c5: Status 404 returned error can't find the container with id 7155889f52e4d135a6df02962d2ba6c3820541b5907b4ac972d5f3eee670a4c5 Oct 30 02:59:18.553473 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:18.553398 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" event={"ID":"b2f0a2df-aef0-4a8a-8465-959dc18036c8","Type":"ContainerStarted","Data":"7155889f52e4d135a6df02962d2ba6c3820541b5907b4ac972d5f3eee670a4c5"} Oct 30 02:59:19.560621 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:19.560579 5035 generic.go:356] "Generic (PLEG): container finished" podID="b2f0a2df-aef0-4a8a-8465-959dc18036c8" containerID="ca2268096da3e953f83310d329659397a7562cc8ab00d02eccf4384b2d20e8e2" exitCode=0 Oct 30 02:59:19.561059 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:19.560653 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" event={"ID":"b2f0a2df-aef0-4a8a-8465-959dc18036c8","Type":"ContainerDied","Data":"ca2268096da3e953f83310d329659397a7562cc8ab00d02eccf4384b2d20e8e2"} Oct 30 02:59:20.713212 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.713179 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:20.870887 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.870676 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-openshift-ca\") pod \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " Oct 30 02:59:20.871342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.870898 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f0a2df-aef0-4a8a-8465-959dc18036c8-kube-api-access-gfzrj\" (UniqueName: \"kubernetes.io/projected/b2f0a2df-aef0-4a8a-8465-959dc18036c8-kube-api-access-gfzrj\") pod \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " Oct 30 02:59:20.871342 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.871284 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-xks-ca\") pod \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\" (UID: \"b2f0a2df-aef0-4a8a-8465-959dc18036c8\") " Oct 30 02:59:20.871610 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.871571 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-openshift-ca" pod "b2f0a2df-aef0-4a8a-8465-959dc18036c8" (UID: "b2f0a2df-aef0-4a8a-8465-959dc18036c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:59:20.872077 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.872039 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:20.875155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.875077 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-xks-ca" pod "b2f0a2df-aef0-4a8a-8465-959dc18036c8" (UID: "b2f0a2df-aef0-4a8a-8465-959dc18036c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:59:20.875155 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.875134 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f0a2df-aef0-4a8a-8465-959dc18036c8-kube-api-access-gfzrj" pod "b2f0a2df-aef0-4a8a-8465-959dc18036c8" (UID: "b2f0a2df-aef0-4a8a-8465-959dc18036c8"). InnerVolumeSpecName "kube-api-access-gfzrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:59:20.974320 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.974170 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f0a2df-aef0-4a8a-8465-959dc18036c8-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:20.974320 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:20.974294 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfzrj\" (UniqueName: \"kubernetes.io/projected/b2f0a2df-aef0-4a8a-8465-959dc18036c8-kube-api-access-gfzrj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:21.576378 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:21.576345 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" Oct 30 02:59:21.576636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:21.576366 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr" event={"ID":"b2f0a2df-aef0-4a8a-8465-959dc18036c8","Type":"ContainerDied","Data":"7155889f52e4d135a6df02962d2ba6c3820541b5907b4ac972d5f3eee670a4c5"} Oct 30 02:59:21.576636 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:21.576430 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7155889f52e4d135a6df02962d2ba6c3820541b5907b4ac972d5f3eee670a4c5" Oct 30 02:59:32.034972 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:32.034868 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv"] Oct 30 02:59:32.047513 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:32.047431 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rpmfv"] Oct 30 02:59:33.665446 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:33.665347 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658" path="/var/lib/kubelet/pods/5ba7b1ea-b0fc-4ca0-8a6b-6a89652ed658/volumes" Oct 30 02:59:35.056751 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.056708 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g"] Oct 30 02:59:35.057287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.057179 5035 cpu_manager.go:414] "containerMap: removing container" podUID="b2f0a2df-aef0-4a8a-8465-959dc18036c8" containerName="revoke-keys" Oct 30 02:59:35.057287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.057191 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f0a2df-aef0-4a8a-8465-959dc18036c8" containerName="revoke-keys" Oct 30 02:59:35.057287 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.057287 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f0a2df-aef0-4a8a-8465-959dc18036c8" containerName="revoke-keys" Oct 30 02:59:35.060866 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.060837 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.087397 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.087356 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:59:35.087645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.087404 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:59:35.087645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.087435 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:59:35.087645 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.087435 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:59:35.093072 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.092961 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g"] Oct 30 02:59:35.166005 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.165957 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-57g6g\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.166218 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.166105 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-57g6g\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.166218 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.166156 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd8g\" (UniqueName: \"kubernetes.io/projected/374f9c27-0f52-4f56-abde-233714fa6e04-kube-api-access-gvd8g\") pod \"maas-api-revoke-sub-a4aef7d7-57g6g\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.267293 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.267249 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-57g6g\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.267494 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.267361 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-57g6g\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.267494 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.267417 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd8g\" (UniqueName: \"kubernetes.io/projected/374f9c27-0f52-4f56-abde-233714fa6e04-kube-api-access-gvd8g\") pod \"maas-api-revoke-sub-a4aef7d7-57g6g\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.268129 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.268104 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-57g6g\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.269834 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.269814 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-57g6g\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.293202 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.293162 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd8g\" (UniqueName: \"kubernetes.io/projected/374f9c27-0f52-4f56-abde-233714fa6e04-kube-api-access-gvd8g\") pod \"maas-api-revoke-sub-a4aef7d7-57g6g\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.375859 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.375618 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:35.686164 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.686130 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g"] Oct 30 02:59:35.688919 ip-10-0-142-110 kubenswrapper[5035]: W1030 02:59:35.688876 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374f9c27_0f52_4f56_abde_233714fa6e04.slice/crio-35c8fb281ce857d04eca9a8bdf7275773aa610f516d15eafabad08aa7b293aaf WatchSource:0}: Error finding container 35c8fb281ce857d04eca9a8bdf7275773aa610f516d15eafabad08aa7b293aaf: Status 404 returned error can't find the container with id 35c8fb281ce857d04eca9a8bdf7275773aa610f516d15eafabad08aa7b293aaf Oct 30 02:59:35.753904 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:35.753863 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" event={"ID":"374f9c27-0f52-4f56-abde-233714fa6e04","Type":"ContainerStarted","Data":"35c8fb281ce857d04eca9a8bdf7275773aa610f516d15eafabad08aa7b293aaf"} Oct 30 02:59:36.762272 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:36.762228 5035 generic.go:356] "Generic (PLEG): container finished" podID="374f9c27-0f52-4f56-abde-233714fa6e04" containerID="55feda34144455de99f431a3e3302b4dfa91fcff19061d6514a026f5dc813724" exitCode=0 Oct 30 02:59:36.762770 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:36.762311 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" event={"ID":"374f9c27-0f52-4f56-abde-233714fa6e04","Type":"ContainerDied","Data":"55feda34144455de99f431a3e3302b4dfa91fcff19061d6514a026f5dc813724"} Oct 30 02:59:37.966790 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:37.966658 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:38.099655 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.099523 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-xks-ca\") pod \"374f9c27-0f52-4f56-abde-233714fa6e04\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " Oct 30 02:59:38.099835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.099758 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374f9c27-0f52-4f56-abde-233714fa6e04-kube-api-access-gvd8g\" (UniqueName: \"kubernetes.io/projected/374f9c27-0f52-4f56-abde-233714fa6e04-kube-api-access-gvd8g\") pod \"374f9c27-0f52-4f56-abde-233714fa6e04\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " Oct 30 02:59:38.099835 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.099791 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-openshift-ca\") pod \"374f9c27-0f52-4f56-abde-233714fa6e04\" (UID: \"374f9c27-0f52-4f56-abde-233714fa6e04\") " Oct 30 02:59:38.100217 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.100189 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-openshift-ca" pod "374f9c27-0f52-4f56-abde-233714fa6e04" (UID: "374f9c27-0f52-4f56-abde-233714fa6e04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:59:38.101757 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.101732 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-xks-ca" pod "374f9c27-0f52-4f56-abde-233714fa6e04" (UID: "374f9c27-0f52-4f56-abde-233714fa6e04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:59:38.101842 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.101820 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374f9c27-0f52-4f56-abde-233714fa6e04-kube-api-access-gvd8g" pod "374f9c27-0f52-4f56-abde-233714fa6e04" (UID: "374f9c27-0f52-4f56-abde-233714fa6e04"). InnerVolumeSpecName "kube-api-access-gvd8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:59:38.200623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.200577 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:38.200623 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.200617 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvd8g\" (UniqueName: \"kubernetes.io/projected/374f9c27-0f52-4f56-abde-233714fa6e04-kube-api-access-gvd8g\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:38.200857 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.200644 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/374f9c27-0f52-4f56-abde-233714fa6e04-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:38.780987 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.780933 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" Oct 30 02:59:38.781297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.780934 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g" event={"ID":"374f9c27-0f52-4f56-abde-233714fa6e04","Type":"ContainerDied","Data":"35c8fb281ce857d04eca9a8bdf7275773aa610f516d15eafabad08aa7b293aaf"} Oct 30 02:59:38.781297 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:38.781100 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35c8fb281ce857d04eca9a8bdf7275773aa610f516d15eafabad08aa7b293aaf" Oct 30 02:59:47.628994 ip-10-0-142-110 kubenswrapper[5035]: E1030 02:59:47.628949 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 02:59:49.035216 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:49.035158 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5"] Oct 30 02:59:49.048510 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:49.048429 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7db290-tszn5"] Oct 30 02:59:49.642066 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:49.642028 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42a265e-d25e-41ae-82f5-4e1880a0ca29" path="/var/lib/kubelet/pods/e42a265e-d25e-41ae-82f5-4e1880a0ca29/volumes" Oct 30 02:59:52.134073 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.134021 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr"] Oct 30 02:59:52.135507 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.135469 5035 cpu_manager.go:414] "containerMap: removing container" podUID="374f9c27-0f52-4f56-abde-233714fa6e04" containerName="revoke-keys" Oct 30 02:59:52.135735 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.135718 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="374f9c27-0f52-4f56-abde-233714fa6e04" containerName="revoke-keys" Oct 30 02:59:52.136069 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.136050 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="374f9c27-0f52-4f56-abde-233714fa6e04" containerName="revoke-keys" Oct 30 02:59:52.149553 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.148231 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.154437 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.154401 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr"] Oct 30 02:59:52.165008 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.164950 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:59:52.181156 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.181097 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:59:52.181590 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.181563 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:59:52.181867 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.181851 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:59:52.320804 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.320694 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2bjvr\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.340868 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.340807 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2bjvr\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.341308 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.341278 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhh7q\" (UniqueName: \"kubernetes.io/projected/136ea7d6-8486-4f60-958a-3752c3655ef2-kube-api-access-vhh7q\") pod \"maas-api-revoke-sub-79805ca9-2bjvr\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.447268 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.447143 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhh7q\" (UniqueName: \"kubernetes.io/projected/136ea7d6-8486-4f60-958a-3752c3655ef2-kube-api-access-vhh7q\") pod \"maas-api-revoke-sub-79805ca9-2bjvr\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.448625 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.448595 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2bjvr\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.449742 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.449719 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2bjvr\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.456796 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.449546 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2bjvr\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.459920 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.459847 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2bjvr\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.491569 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.491463 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhh7q\" (UniqueName: \"kubernetes.io/projected/136ea7d6-8486-4f60-958a-3752c3655ef2-kube-api-access-vhh7q\") pod \"maas-api-revoke-sub-79805ca9-2bjvr\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:52.573565 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:52.571721 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:53.555558 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:53.538978 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr"] Oct 30 02:59:53.948026 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:53.947983 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" event={"ID":"136ea7d6-8486-4f60-958a-3752c3655ef2","Type":"ContainerStarted","Data":"e87dab59c0bb026f4db9bcaa2be95277d010ec596ffe5f999cb69a5da04ac2db"} Oct 30 02:59:54.956372 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:54.956334 5035 generic.go:356] "Generic (PLEG): container finished" podID="136ea7d6-8486-4f60-958a-3752c3655ef2" containerID="02d78aa719c697edd38683bedfce5567e2931c2f36797b295b97c01cc6738bf0" exitCode=0 Oct 30 02:59:54.956863 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:54.956389 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" event={"ID":"136ea7d6-8486-4f60-958a-3752c3655ef2","Type":"ContainerDied","Data":"02d78aa719c697edd38683bedfce5567e2931c2f36797b295b97c01cc6738bf0"} Oct 30 02:59:56.110012 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.109978 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:56.163771 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.163733 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136ea7d6-8486-4f60-958a-3752c3655ef2-kube-api-access-vhh7q\" (UniqueName: \"kubernetes.io/projected/136ea7d6-8486-4f60-958a-3752c3655ef2-kube-api-access-vhh7q\") pod \"136ea7d6-8486-4f60-958a-3752c3655ef2\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " Oct 30 02:59:56.164006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.163836 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-xks-ca\") pod \"136ea7d6-8486-4f60-958a-3752c3655ef2\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " Oct 30 02:59:56.164006 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.163936 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-openshift-ca\") pod \"136ea7d6-8486-4f60-958a-3752c3655ef2\" (UID: \"136ea7d6-8486-4f60-958a-3752c3655ef2\") " Oct 30 02:59:56.164353 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.164324 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-openshift-ca" pod "136ea7d6-8486-4f60-958a-3752c3655ef2" (UID: "136ea7d6-8486-4f60-958a-3752c3655ef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:59:56.166099 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.166071 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136ea7d6-8486-4f60-958a-3752c3655ef2-kube-api-access-vhh7q" pod "136ea7d6-8486-4f60-958a-3752c3655ef2" (UID: "136ea7d6-8486-4f60-958a-3752c3655ef2"). InnerVolumeSpecName "kube-api-access-vhh7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:59:56.166493 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.166466 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-xks-ca" pod "136ea7d6-8486-4f60-958a-3752c3655ef2" (UID: "136ea7d6-8486-4f60-958a-3752c3655ef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:59:56.265639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.265506 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:56.265639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.265583 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136ea7d6-8486-4f60-958a-3752c3655ef2-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:56.265639 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.265600 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhh7q\" (UniqueName: \"kubernetes.io/projected/136ea7d6-8486-4f60-958a-3752c3655ef2-kube-api-access-vhh7q\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 02:59:56.974050 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.974021 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" Oct 30 02:59:56.974248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.974060 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr" event={"ID":"136ea7d6-8486-4f60-958a-3752c3655ef2","Type":"ContainerDied","Data":"e87dab59c0bb026f4db9bcaa2be95277d010ec596ffe5f999cb69a5da04ac2db"} Oct 30 02:59:56.974248 ip-10-0-142-110 kubenswrapper[5035]: I1030 02:59:56.974106 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e87dab59c0bb026f4db9bcaa2be95277d010ec596ffe5f999cb69a5da04ac2db" Oct 30 03:00:00.152209 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.152168 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc"] Oct 30 03:00:00.155873 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.153153 5035 cpu_manager.go:414] "containerMap: removing container" podUID="136ea7d6-8486-4f60-958a-3752c3655ef2" containerName="revoke-keys" Oct 30 03:00:00.155873 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.153192 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136ea7d6-8486-4f60-958a-3752c3655ef2" containerName="revoke-keys" Oct 30 03:00:00.155873 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.153600 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="136ea7d6-8486-4f60-958a-3752c3655ef2" containerName="revoke-keys" Oct 30 03:00:00.164647 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.164613 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:00.176868 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.176833 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc"] Oct 30 03:00:00.184338 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.184301 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:00:00.194097 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.194059 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:00:00.194292 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.194160 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:00:00.312489 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.312439 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp79k\" (UniqueName: \"kubernetes.io/projected/35601646-5413-4ea6-a6d0-8262266f1b90-kube-api-access-pp79k\") pod \"maas-api-key-cleanup-29888820-5nwbc\" (UID: \"35601646-5413-4ea6-a6d0-8262266f1b90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:00.312726 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.312635 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35601646-5413-4ea6-a6d0-8262266f1b90-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888820-5nwbc\" (UID: \"35601646-5413-4ea6-a6d0-8262266f1b90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:00.413865 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.413765 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp79k\" (UniqueName: \"kubernetes.io/projected/35601646-5413-4ea6-a6d0-8262266f1b90-kube-api-access-pp79k\") pod \"maas-api-key-cleanup-29888820-5nwbc\" (UID: \"35601646-5413-4ea6-a6d0-8262266f1b90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:00.413865 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.413865 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35601646-5413-4ea6-a6d0-8262266f1b90-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888820-5nwbc\" (UID: \"35601646-5413-4ea6-a6d0-8262266f1b90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:00.414523 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.414499 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35601646-5413-4ea6-a6d0-8262266f1b90-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888820-5nwbc\" (UID: \"35601646-5413-4ea6-a6d0-8262266f1b90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:00.437812 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.437774 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp79k\" (UniqueName: \"kubernetes.io/projected/35601646-5413-4ea6-a6d0-8262266f1b90-kube-api-access-pp79k\") pod \"maas-api-key-cleanup-29888820-5nwbc\" (UID: \"35601646-5413-4ea6-a6d0-8262266f1b90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:00.482318 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.482273 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:00.790982 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:00.790953 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc"] Oct 30 03:00:00.792704 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:00:00.792673 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35601646_5413_4ea6_a6d0_8262266f1b90.slice/crio-4e9a8b58ff99977055775057d0aa057992630119e0c1703da4e9cd56c9dff146 WatchSource:0}: Error finding container 4e9a8b58ff99977055775057d0aa057992630119e0c1703da4e9cd56c9dff146: Status 404 returned error can't find the container with id 4e9a8b58ff99977055775057d0aa057992630119e0c1703da4e9cd56c9dff146 Oct 30 03:00:01.000558 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:01.000504 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" event={"ID":"35601646-5413-4ea6-a6d0-8262266f1b90","Type":"ContainerStarted","Data":"4e9a8b58ff99977055775057d0aa057992630119e0c1703da4e9cd56c9dff146"} Oct 30 03:00:02.008137 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:02.008096 5035 generic.go:356] "Generic (PLEG): container finished" podID="35601646-5413-4ea6-a6d0-8262266f1b90" containerID="10e6125b0260f1c48853ba73423337d7d9c22eb46b6a0803912c1f93877fccb7" exitCode=60 Oct 30 03:00:02.008574 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:02.008185 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" event={"ID":"35601646-5413-4ea6-a6d0-8262266f1b90","Type":"ContainerDied","Data":"10e6125b0260f1c48853ba73423337d7d9c22eb46b6a0803912c1f93877fccb7"} Oct 30 03:00:02.008574 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:02.008417 5035 scope.go:122] "RemoveContainer" containerID="10e6125b0260f1c48853ba73423337d7d9c22eb46b6a0803912c1f93877fccb7" Oct 30 03:00:03.015512 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:03.015472 5035 generic.go:356] "Generic (PLEG): container finished" podID="35601646-5413-4ea6-a6d0-8262266f1b90" containerID="1508930de9cfde1a547b3537a1ebf092f855bbaa19e74f1330cb28272eda5429" exitCode=60 Oct 30 03:00:03.016018 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:03.015558 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" event={"ID":"35601646-5413-4ea6-a6d0-8262266f1b90","Type":"ContainerDied","Data":"1508930de9cfde1a547b3537a1ebf092f855bbaa19e74f1330cb28272eda5429"} Oct 30 03:00:03.016018 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:03.015630 5035 scope.go:122] "RemoveContainer" containerID="10e6125b0260f1c48853ba73423337d7d9c22eb46b6a0803912c1f93877fccb7" Oct 30 03:00:03.016018 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:03.015875 5035 scope.go:122] "RemoveContainer" containerID="1508930de9cfde1a547b3537a1ebf092f855bbaa19e74f1330cb28272eda5429" Oct 30 03:00:03.016203 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:00:03.016182 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888820-5nwbc_odh-ai-gateway-infra(35601646-5413-4ea6-a6d0-8262266f1b90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" Oct 30 03:00:04.023960 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:04.023925 5035 scope.go:122] "RemoveContainer" containerID="1508930de9cfde1a547b3537a1ebf092f855bbaa19e74f1330cb28272eda5429" Oct 30 03:00:04.024372 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:00:04.024191 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888820-5nwbc_odh-ai-gateway-infra(35601646-5413-4ea6-a6d0-8262266f1b90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" Oct 30 03:00:16.635142 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:16.635106 5035 scope.go:122] "RemoveContainer" containerID="1508930de9cfde1a547b3537a1ebf092f855bbaa19e74f1330cb28272eda5429" Oct 30 03:00:17.478293 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:17.478248 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" event={"ID":"35601646-5413-4ea6-a6d0-8262266f1b90","Type":"ContainerStarted","Data":"2f39d687d83a0b9fa5aee4298a6483f100b4260e152c324f9eaafaa11bf87493"} Oct 30 03:00:17.520658 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:17.520574 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" podStartSLOduration=17.520547629 podStartE2EDuration="17.520547629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:17.518550974 +0000 UTC m=+5496.021483198" watchObservedRunningTime="2026-10-30 03:00:17.520547629 +0000 UTC m=+5496.023479850" Oct 30 03:00:17.629786 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:00:17.629746 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:00:18.486439 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:18.486403 5035 generic.go:356] "Generic (PLEG): container finished" podID="35601646-5413-4ea6-a6d0-8262266f1b90" containerID="2f39d687d83a0b9fa5aee4298a6483f100b4260e152c324f9eaafaa11bf87493" exitCode=60 Oct 30 03:00:18.487009 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:18.486457 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" event={"ID":"35601646-5413-4ea6-a6d0-8262266f1b90","Type":"ContainerDied","Data":"2f39d687d83a0b9fa5aee4298a6483f100b4260e152c324f9eaafaa11bf87493"} Oct 30 03:00:18.487009 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:18.486505 5035 scope.go:122] "RemoveContainer" containerID="1508930de9cfde1a547b3537a1ebf092f855bbaa19e74f1330cb28272eda5429" Oct 30 03:00:18.487009 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:18.486941 5035 scope.go:122] "RemoveContainer" containerID="2f39d687d83a0b9fa5aee4298a6483f100b4260e152c324f9eaafaa11bf87493" Oct 30 03:00:18.487275 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:00:18.487249 5035 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888820-5nwbc_odh-ai-gateway-infra(35601646-5413-4ea6-a6d0-8262266f1b90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" Oct 30 03:00:18.540649 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:18.540605 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc"] Oct 30 03:00:19.654464 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:19.654427 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:19.706216 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:19.706175 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35601646-5413-4ea6-a6d0-8262266f1b90-kube-api-access-pp79k\" (UniqueName: \"kubernetes.io/projected/35601646-5413-4ea6-a6d0-8262266f1b90-kube-api-access-pp79k\") pod \"35601646-5413-4ea6-a6d0-8262266f1b90\" (UID: \"35601646-5413-4ea6-a6d0-8262266f1b90\") " Oct 30 03:00:19.706410 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:19.706273 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35601646-5413-4ea6-a6d0-8262266f1b90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35601646-5413-4ea6-a6d0-8262266f1b90-maas-api-service-ca\") pod \"35601646-5413-4ea6-a6d0-8262266f1b90\" (UID: \"35601646-5413-4ea6-a6d0-8262266f1b90\") " Oct 30 03:00:19.706686 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:19.706657 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35601646-5413-4ea6-a6d0-8262266f1b90-maas-api-service-ca" pod "35601646-5413-4ea6-a6d0-8262266f1b90" (UID: "35601646-5413-4ea6-a6d0-8262266f1b90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:00:19.706841 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:19.706823 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35601646-5413-4ea6-a6d0-8262266f1b90-maas-api-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:00:19.708416 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:19.708390 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35601646-5413-4ea6-a6d0-8262266f1b90-kube-api-access-pp79k" pod "35601646-5413-4ea6-a6d0-8262266f1b90" (UID: "35601646-5413-4ea6-a6d0-8262266f1b90"). InnerVolumeSpecName "kube-api-access-pp79k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:00:19.808036 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:19.807932 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp79k\" (UniqueName: \"kubernetes.io/projected/35601646-5413-4ea6-a6d0-8262266f1b90-kube-api-access-pp79k\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:00:20.501247 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:20.501209 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" Oct 30 03:00:20.501424 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:20.501252 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc" event={"ID":"35601646-5413-4ea6-a6d0-8262266f1b90","Type":"ContainerDied","Data":"4e9a8b58ff99977055775057d0aa057992630119e0c1703da4e9cd56c9dff146"} Oct 30 03:00:20.501424 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:20.501316 5035 scope.go:122] "RemoveContainer" containerID="2f39d687d83a0b9fa5aee4298a6483f100b4260e152c324f9eaafaa11bf87493" Oct 30 03:00:20.552083 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:20.552050 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc"] Oct 30 03:00:20.561419 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:20.561384 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888820-5nwbc"] Oct 30 03:00:21.649512 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:21.649466 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" path="/var/lib/kubelet/pods/35601646-5413-4ea6-a6d0-8262266f1b90/volumes" Oct 30 03:00:31.494739 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:31.494676 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:00:31.495696 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:31.495656 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:00:31.538697 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:31.538649 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:00:31.539027 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:31.538993 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:00:35.940285 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.940242 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t"] Oct 30 03:00:35.940769 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.940753 5035 cpu_manager.go:414] "containerMap: removing container" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" containerName="cleanup" Oct 30 03:00:35.940817 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.940770 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" containerName="cleanup" Oct 30 03:00:35.940817 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.940780 5035 cpu_manager.go:414] "containerMap: removing container" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" containerName="cleanup" Oct 30 03:00:35.940817 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.940785 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" containerName="cleanup" Oct 30 03:00:35.940922 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.940863 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" containerName="cleanup" Oct 30 03:00:35.940922 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.940872 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" containerName="cleanup" Oct 30 03:00:35.940922 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.940878 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" containerName="cleanup" Oct 30 03:00:35.944144 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.944111 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:35.973744 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.973707 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:00:35.974005 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.973935 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:00:35.976847 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.976813 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:00:35.977024 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.976844 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:00:35.977024 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:35.976829 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t"] Oct 30 03:00:36.029818 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.029781 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7tr\" (UniqueName: \"kubernetes.io/projected/cc193d7e-5bc2-4622-ae61-a34e3107f923-kube-api-access-wn7tr\") pod \"maas-api-revoke-sub-3d4156f1-9bh7t\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.030028 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.029899 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9bh7t\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.030028 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.029941 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9bh7t\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.131428 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.131381 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7tr\" (UniqueName: \"kubernetes.io/projected/cc193d7e-5bc2-4622-ae61-a34e3107f923-kube-api-access-wn7tr\") pod \"maas-api-revoke-sub-3d4156f1-9bh7t\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.131705 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.131468 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9bh7t\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.131705 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.131507 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9bh7t\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.132196 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.132171 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9bh7t\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.134095 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.134068 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9bh7t\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.154893 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.154855 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7tr\" (UniqueName: \"kubernetes.io/projected/cc193d7e-5bc2-4622-ae61-a34e3107f923-kube-api-access-wn7tr\") pod \"maas-api-revoke-sub-3d4156f1-9bh7t\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.260718 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.260672 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:36.565295 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.565259 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t"] Oct 30 03:00:36.567022 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:00:36.566984 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc193d7e_5bc2_4622_ae61_a34e3107f923.slice/crio-da385ed100060f80de9c448913300b11e00b07edb6d160df72c68b1c2266d332 WatchSource:0}: Error finding container da385ed100060f80de9c448913300b11e00b07edb6d160df72c68b1c2266d332: Status 404 returned error can't find the container with id da385ed100060f80de9c448913300b11e00b07edb6d160df72c68b1c2266d332 Oct 30 03:00:36.653598 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:36.653556 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" event={"ID":"cc193d7e-5bc2-4622-ae61-a34e3107f923","Type":"ContainerStarted","Data":"da385ed100060f80de9c448913300b11e00b07edb6d160df72c68b1c2266d332"} Oct 30 03:00:37.661259 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:37.661226 5035 generic.go:356] "Generic (PLEG): container finished" podID="cc193d7e-5bc2-4622-ae61-a34e3107f923" containerID="01ec9740686f21be2f1bd5c8395f07d2a9ceff0fc87d689422fd779155e35176" exitCode=0 Oct 30 03:00:37.661689 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:37.661309 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" event={"ID":"cc193d7e-5bc2-4622-ae61-a34e3107f923","Type":"ContainerDied","Data":"01ec9740686f21be2f1bd5c8395f07d2a9ceff0fc87d689422fd779155e35176"} Oct 30 03:00:38.817989 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.817960 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:38.862502 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.862463 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-openshift-ca\") pod \"cc193d7e-5bc2-4622-ae61-a34e3107f923\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " Oct 30 03:00:38.862728 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.862633 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-xks-ca\") pod \"cc193d7e-5bc2-4622-ae61-a34e3107f923\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " Oct 30 03:00:38.862728 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.862706 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc193d7e-5bc2-4622-ae61-a34e3107f923-kube-api-access-wn7tr\" (UniqueName: \"kubernetes.io/projected/cc193d7e-5bc2-4622-ae61-a34e3107f923-kube-api-access-wn7tr\") pod \"cc193d7e-5bc2-4622-ae61-a34e3107f923\" (UID: \"cc193d7e-5bc2-4622-ae61-a34e3107f923\") " Oct 30 03:00:38.862929 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.862901 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-openshift-ca" pod "cc193d7e-5bc2-4622-ae61-a34e3107f923" (UID: "cc193d7e-5bc2-4622-ae61-a34e3107f923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:00:38.863101 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.863084 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:00:38.864914 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.864878 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-xks-ca" pod "cc193d7e-5bc2-4622-ae61-a34e3107f923" (UID: "cc193d7e-5bc2-4622-ae61-a34e3107f923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:00:38.865037 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.864919 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc193d7e-5bc2-4622-ae61-a34e3107f923-kube-api-access-wn7tr" pod "cc193d7e-5bc2-4622-ae61-a34e3107f923" (UID: "cc193d7e-5bc2-4622-ae61-a34e3107f923"). InnerVolumeSpecName "kube-api-access-wn7tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:00:38.964585 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.964550 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc193d7e-5bc2-4622-ae61-a34e3107f923-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:00:38.964585 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:38.964583 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7tr\" (UniqueName: \"kubernetes.io/projected/cc193d7e-5bc2-4622-ae61-a34e3107f923-kube-api-access-wn7tr\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:00:39.676181 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:39.676135 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" event={"ID":"cc193d7e-5bc2-4622-ae61-a34e3107f923","Type":"ContainerDied","Data":"da385ed100060f80de9c448913300b11e00b07edb6d160df72c68b1c2266d332"} Oct 30 03:00:39.676181 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:39.676165 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t" Oct 30 03:00:39.676414 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:39.676200 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da385ed100060f80de9c448913300b11e00b07edb6d160df72c68b1c2266d332" Oct 30 03:00:45.438017 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:45.437916 5035 scope.go:122] "RemoveContainer" containerID="927d11d9b11626b7bd6aa04be65a4f79b77dfb7aa50f2f04a36be28a0e51a524" Oct 30 03:00:45.462948 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:00:45.462874 5035 scope.go:122] "RemoveContainer" containerID="4572e4c9bb6550c0f5b5e3c09f9ddc4bdc1a5b9fb743feda31c4757083cd3716" Oct 30 03:00:47.628201 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:00:47.628139 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:01:01.517827 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:01.517688 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:01:01.518566 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:01.518500 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:01:01.544695 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:01.544643 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:01:01.545036 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:01.545001 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:01:05.482804 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.482635 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9"] Oct 30 03:01:05.484349 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.484000 5035 cpu_manager.go:414] "containerMap: removing container" podUID="cc193d7e-5bc2-4622-ae61-a34e3107f923" containerName="revoke-keys" Oct 30 03:01:05.484349 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.484047 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc193d7e-5bc2-4622-ae61-a34e3107f923" containerName="revoke-keys" Oct 30 03:01:05.484349 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.484147 5035 cpu_manager.go:414] "containerMap: removing container" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" containerName="cleanup" Oct 30 03:01:05.484349 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.484156 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35601646-5413-4ea6-a6d0-8262266f1b90" containerName="cleanup" Oct 30 03:01:05.484349 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.484311 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc193d7e-5bc2-4622-ae61-a34e3107f923" containerName="revoke-keys" Oct 30 03:01:05.490667 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.490626 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.506770 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.506731 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9"] Oct 30 03:01:05.516070 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.514956 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:01:05.516070 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.514994 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:01:05.516070 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.514996 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:01:05.517023 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.516994 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:01:05.529509 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.529468 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjcb2\" (UniqueName: \"kubernetes.io/projected/19efc19a-a22c-4560-b974-bb2a1e3bfb79-kube-api-access-zjcb2\") pod \"maas-api-revoke-sub-e2fc9c5b-9h4p9\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.529762 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.529516 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9h4p9\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.529762 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.529734 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9h4p9\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.631038 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.630993 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjcb2\" (UniqueName: \"kubernetes.io/projected/19efc19a-a22c-4560-b974-bb2a1e3bfb79-kube-api-access-zjcb2\") pod \"maas-api-revoke-sub-e2fc9c5b-9h4p9\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.631038 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.631039 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9h4p9\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.631298 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.631114 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9h4p9\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.631765 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.631742 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9h4p9\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.633649 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.633624 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9h4p9\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.654635 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.654599 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjcb2\" (UniqueName: \"kubernetes.io/projected/19efc19a-a22c-4560-b974-bb2a1e3bfb79-kube-api-access-zjcb2\") pod \"maas-api-revoke-sub-e2fc9c5b-9h4p9\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:05.806493 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:05.806383 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:06.113883 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:06.113848 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9"] Oct 30 03:01:06.115705 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:01:06.115677 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19efc19a_a22c_4560_b974_bb2a1e3bfb79.slice/crio-a0fa7ee0b55c35b1a7a4f591b5311714a9c21d38b983f7647ae93ad8aeb1aa0f WatchSource:0}: Error finding container a0fa7ee0b55c35b1a7a4f591b5311714a9c21d38b983f7647ae93ad8aeb1aa0f: Status 404 returned error can't find the container with id a0fa7ee0b55c35b1a7a4f591b5311714a9c21d38b983f7647ae93ad8aeb1aa0f Oct 30 03:01:06.892191 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:06.892149 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" event={"ID":"19efc19a-a22c-4560-b974-bb2a1e3bfb79","Type":"ContainerStarted","Data":"d1465217979a5c946ff96d3d53f66918ef42c6d576b4e52ac3cf96b6a1976802"} Oct 30 03:01:06.892683 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:06.892206 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" event={"ID":"19efc19a-a22c-4560-b974-bb2a1e3bfb79","Type":"ContainerStarted","Data":"a0fa7ee0b55c35b1a7a4f591b5311714a9c21d38b983f7647ae93ad8aeb1aa0f"} Oct 30 03:01:06.933953 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:06.933887 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" podStartSLOduration=1.933868889 podStartE2EDuration="1.933868889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:06.932495657 +0000 UTC m=+5545.435427877" watchObservedRunningTime="2026-10-30 03:01:06.933868889 +0000 UTC m=+5545.436801110" Oct 30 03:01:07.901148 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:07.901083 5035 generic.go:356] "Generic (PLEG): container finished" podID="19efc19a-a22c-4560-b974-bb2a1e3bfb79" containerID="d1465217979a5c946ff96d3d53f66918ef42c6d576b4e52ac3cf96b6a1976802" exitCode=0 Oct 30 03:01:07.901801 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:07.901162 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" event={"ID":"19efc19a-a22c-4560-b974-bb2a1e3bfb79","Type":"ContainerDied","Data":"d1465217979a5c946ff96d3d53f66918ef42c6d576b4e52ac3cf96b6a1976802"} Oct 30 03:01:09.067095 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.067062 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:09.168815 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.168717 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-xks-ca\") pod \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " Oct 30 03:01:09.168815 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.168780 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19efc19a-a22c-4560-b974-bb2a1e3bfb79-kube-api-access-zjcb2\" (UniqueName: \"kubernetes.io/projected/19efc19a-a22c-4560-b974-bb2a1e3bfb79-kube-api-access-zjcb2\") pod \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " Oct 30 03:01:09.169037 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.168822 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-openshift-ca\") pod \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\" (UID: \"19efc19a-a22c-4560-b974-bb2a1e3bfb79\") " Oct 30 03:01:09.169291 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.169255 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-openshift-ca" pod "19efc19a-a22c-4560-b974-bb2a1e3bfb79" (UID: "19efc19a-a22c-4560-b974-bb2a1e3bfb79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:01:09.170898 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.170871 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-xks-ca" pod "19efc19a-a22c-4560-b974-bb2a1e3bfb79" (UID: "19efc19a-a22c-4560-b974-bb2a1e3bfb79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:01:09.171030 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.171013 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19efc19a-a22c-4560-b974-bb2a1e3bfb79-kube-api-access-zjcb2" pod "19efc19a-a22c-4560-b974-bb2a1e3bfb79" (UID: "19efc19a-a22c-4560-b974-bb2a1e3bfb79"). InnerVolumeSpecName "kube-api-access-zjcb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:01:09.270978 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.270895 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:01:09.270978 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.270965 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjcb2\" (UniqueName: \"kubernetes.io/projected/19efc19a-a22c-4560-b974-bb2a1e3bfb79-kube-api-access-zjcb2\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:01:09.270978 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.270980 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19efc19a-a22c-4560-b974-bb2a1e3bfb79-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:01:09.923028 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.922805 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" event={"ID":"19efc19a-a22c-4560-b974-bb2a1e3bfb79","Type":"ContainerDied","Data":"a0fa7ee0b55c35b1a7a4f591b5311714a9c21d38b983f7647ae93ad8aeb1aa0f"} Oct 30 03:01:09.923028 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.922935 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0fa7ee0b55c35b1a7a4f591b5311714a9c21d38b983f7647ae93ad8aeb1aa0f" Oct 30 03:01:09.923028 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:09.922845 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9" Oct 30 03:01:15.812843 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.812797 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm"] Oct 30 03:01:15.813498 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.813476 5035 cpu_manager.go:414] "containerMap: removing container" podUID="19efc19a-a22c-4560-b974-bb2a1e3bfb79" containerName="revoke-keys" Oct 30 03:01:15.813566 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.813501 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19efc19a-a22c-4560-b974-bb2a1e3bfb79" containerName="revoke-keys" Oct 30 03:01:15.813699 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.813685 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="19efc19a-a22c-4560-b974-bb2a1e3bfb79" containerName="revoke-keys" Oct 30 03:01:15.817375 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.817350 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:15.835777 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.835742 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm"] Oct 30 03:01:15.844518 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.844485 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:01:15.844518 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.844504 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:01:15.844756 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.844546 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:01:15.844756 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.844563 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:01:15.952345 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.952299 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cda16d-f399-4592-abac-88d065bac440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-84mpm\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:15.952564 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.952441 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cda16d-f399-4592-abac-88d065bac440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-84mpm\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:15.952564 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:15.952539 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xvj\" (UniqueName: \"kubernetes.io/projected/d1cda16d-f399-4592-abac-88d065bac440-kube-api-access-59xvj\") pod \"maas-api-revoke-sub-5a29efb3-84mpm\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:16.053954 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.053912 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59xvj\" (UniqueName: \"kubernetes.io/projected/d1cda16d-f399-4592-abac-88d065bac440-kube-api-access-59xvj\") pod \"maas-api-revoke-sub-5a29efb3-84mpm\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:16.054189 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.053986 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cda16d-f399-4592-abac-88d065bac440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-84mpm\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:16.054189 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.054108 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cda16d-f399-4592-abac-88d065bac440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-84mpm\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:16.054705 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.054678 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cda16d-f399-4592-abac-88d065bac440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-84mpm\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:16.056414 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.056395 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cda16d-f399-4592-abac-88d065bac440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-84mpm\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:16.079563 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.079474 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xvj\" (UniqueName: \"kubernetes.io/projected/d1cda16d-f399-4592-abac-88d065bac440-kube-api-access-59xvj\") pod \"maas-api-revoke-sub-5a29efb3-84mpm\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:16.130710 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.130670 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:16.458573 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.458543 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm"] Oct 30 03:01:16.460587 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:01:16.460549 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cda16d_f399_4592_abac_88d065bac440.slice/crio-9718584f39338ca2a8f88bbbe4f6a3ac86a69d4832d125abaefb811682177f85 WatchSource:0}: Error finding container 9718584f39338ca2a8f88bbbe4f6a3ac86a69d4832d125abaefb811682177f85: Status 404 returned error can't find the container with id 9718584f39338ca2a8f88bbbe4f6a3ac86a69d4832d125abaefb811682177f85 Oct 30 03:01:16.976087 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.976026 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" event={"ID":"d1cda16d-f399-4592-abac-88d065bac440","Type":"ContainerStarted","Data":"64f304886e6fa566caf42322cef02b89f2ab7e8b6ef2d4e2cab64d3de9f58322"} Oct 30 03:01:16.976087 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:16.976090 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" event={"ID":"d1cda16d-f399-4592-abac-88d065bac440","Type":"ContainerStarted","Data":"9718584f39338ca2a8f88bbbe4f6a3ac86a69d4832d125abaefb811682177f85"} Oct 30 03:01:17.016870 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:17.016809 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" podStartSLOduration=2.016793321 podStartE2EDuration="2.016793321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:17.016099003 +0000 UTC m=+5555.519031235" watchObservedRunningTime="2026-10-30 03:01:17.016793321 +0000 UTC m=+5555.519725541" Oct 30 03:01:17.648522 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:01:17.648426 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:01:17.988615 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:17.988184 5035 generic.go:356] "Generic (PLEG): container finished" podID="d1cda16d-f399-4592-abac-88d065bac440" containerID="64f304886e6fa566caf42322cef02b89f2ab7e8b6ef2d4e2cab64d3de9f58322" exitCode=0 Oct 30 03:01:17.988615 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:17.988282 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" event={"ID":"d1cda16d-f399-4592-abac-88d065bac440","Type":"ContainerDied","Data":"64f304886e6fa566caf42322cef02b89f2ab7e8b6ef2d4e2cab64d3de9f58322"} Oct 30 03:01:20.125201 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.125161 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" event={"ID":"d1cda16d-f399-4592-abac-88d065bac440","Type":"ContainerDied","Data":"9718584f39338ca2a8f88bbbe4f6a3ac86a69d4832d125abaefb811682177f85"} Oct 30 03:01:20.125887 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.125844 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9718584f39338ca2a8f88bbbe4f6a3ac86a69d4832d125abaefb811682177f85" Oct 30 03:01:20.165658 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.165599 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:20.202411 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.198633 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1cda16d-f399-4592-abac-88d065bac440-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cda16d-f399-4592-abac-88d065bac440-maas-api-xks-ca\") pod \"d1cda16d-f399-4592-abac-88d065bac440\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " Oct 30 03:01:20.202411 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.198715 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cda16d-f399-4592-abac-88d065bac440-kube-api-access-59xvj\" (UniqueName: \"kubernetes.io/projected/d1cda16d-f399-4592-abac-88d065bac440-kube-api-access-59xvj\") pod \"d1cda16d-f399-4592-abac-88d065bac440\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " Oct 30 03:01:20.202411 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.198958 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1cda16d-f399-4592-abac-88d065bac440-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cda16d-f399-4592-abac-88d065bac440-maas-api-openshift-ca\") pod \"d1cda16d-f399-4592-abac-88d065bac440\" (UID: \"d1cda16d-f399-4592-abac-88d065bac440\") " Oct 30 03:01:20.202411 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.199763 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cda16d-f399-4592-abac-88d065bac440-maas-api-openshift-ca" pod "d1cda16d-f399-4592-abac-88d065bac440" (UID: "d1cda16d-f399-4592-abac-88d065bac440"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:01:20.202411 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.201442 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1cda16d-f399-4592-abac-88d065bac440-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:01:20.229398 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.229266 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1cda16d-f399-4592-abac-88d065bac440-maas-api-xks-ca" pod "d1cda16d-f399-4592-abac-88d065bac440" (UID: "d1cda16d-f399-4592-abac-88d065bac440"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:01:20.233157 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.233110 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cda16d-f399-4592-abac-88d065bac440-kube-api-access-59xvj" pod "d1cda16d-f399-4592-abac-88d065bac440" (UID: "d1cda16d-f399-4592-abac-88d065bac440"). InnerVolumeSpecName "kube-api-access-59xvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:01:20.302991 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.302795 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1cda16d-f399-4592-abac-88d065bac440-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:01:20.303360 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:20.303329 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59xvj\" (UniqueName: \"kubernetes.io/projected/d1cda16d-f399-4592-abac-88d065bac440-kube-api-access-59xvj\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:01:21.130768 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:21.130730 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm" Oct 30 03:01:31.891914 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.891864 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t"] Oct 30 03:01:31.903264 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.903227 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9bh7t"] Oct 30 03:01:31.935333 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.935290 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv"] Oct 30 03:01:31.935838 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.935822 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d1cda16d-f399-4592-abac-88d065bac440" containerName="revoke-keys" Oct 30 03:01:31.935899 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.935840 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cda16d-f399-4592-abac-88d065bac440" containerName="revoke-keys" Oct 30 03:01:31.935966 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.935956 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cda16d-f399-4592-abac-88d065bac440" containerName="revoke-keys" Oct 30 03:01:31.939402 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.939378 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:31.957613 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.957577 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:01:31.957903 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.957600 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:01:31.958048 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.957662 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:01:31.964960 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.964923 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv"] Oct 30 03:01:31.968640 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:31.968597 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:01:32.032493 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.032441 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskxq\" (UniqueName: \"kubernetes.io/projected/81d0b3a1-0e93-491f-8a51-d071b7165936-kube-api-access-tskxq\") pod \"maas-api-revoke-sub-3d4156f1-l4jjv\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.032733 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.032511 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l4jjv\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.032733 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.032674 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l4jjv\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.133464 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.133408 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l4jjv\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.133701 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.133566 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskxq\" (UniqueName: \"kubernetes.io/projected/81d0b3a1-0e93-491f-8a51-d071b7165936-kube-api-access-tskxq\") pod \"maas-api-revoke-sub-3d4156f1-l4jjv\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.133701 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.133616 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l4jjv\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.134339 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.134317 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l4jjv\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.135892 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.135872 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l4jjv\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.164949 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.164824 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskxq\" (UniqueName: \"kubernetes.io/projected/81d0b3a1-0e93-491f-8a51-d071b7165936-kube-api-access-tskxq\") pod \"maas-api-revoke-sub-3d4156f1-l4jjv\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.265911 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.265840 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:32.580372 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:32.580341 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv"] Oct 30 03:01:32.582619 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:01:32.582588 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d0b3a1_0e93_491f_8a51_d071b7165936.slice/crio-1de9b2d0e12739b28d3c02118132c28b1f45082307709b766da0359d32706421 WatchSource:0}: Error finding container 1de9b2d0e12739b28d3c02118132c28b1f45082307709b766da0359d32706421: Status 404 returned error can't find the container with id 1de9b2d0e12739b28d3c02118132c28b1f45082307709b766da0359d32706421 Oct 30 03:01:33.207737 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:33.207699 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" event={"ID":"81d0b3a1-0e93-491f-8a51-d071b7165936","Type":"ContainerStarted","Data":"bc5f69b182dfdcd253dd053063c494d49549468b07528a2d8f5e870f1aaeb9d9"} Oct 30 03:01:33.208177 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:33.207762 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" event={"ID":"81d0b3a1-0e93-491f-8a51-d071b7165936","Type":"ContainerStarted","Data":"1de9b2d0e12739b28d3c02118132c28b1f45082307709b766da0359d32706421"} Oct 30 03:01:33.248499 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:33.248439 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" podStartSLOduration=2.24842503 podStartE2EDuration="2.24842503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:33.247723457 +0000 UTC m=+5571.750655677" watchObservedRunningTime="2026-10-30 03:01:33.24842503 +0000 UTC m=+5571.751357250" Oct 30 03:01:33.642414 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:33.642373 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc193d7e-5bc2-4622-ae61-a34e3107f923" path="/var/lib/kubelet/pods/cc193d7e-5bc2-4622-ae61-a34e3107f923/volumes" Oct 30 03:01:34.215288 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:34.215248 5035 generic.go:356] "Generic (PLEG): container finished" podID="81d0b3a1-0e93-491f-8a51-d071b7165936" containerID="bc5f69b182dfdcd253dd053063c494d49549468b07528a2d8f5e870f1aaeb9d9" exitCode=0 Oct 30 03:01:34.215732 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:34.215331 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" event={"ID":"81d0b3a1-0e93-491f-8a51-d071b7165936","Type":"ContainerDied","Data":"bc5f69b182dfdcd253dd053063c494d49549468b07528a2d8f5e870f1aaeb9d9"} Oct 30 03:01:35.377212 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.377170 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:35.470374 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.470246 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-openshift-ca\") pod \"81d0b3a1-0e93-491f-8a51-d071b7165936\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " Oct 30 03:01:35.470877 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.470505 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-xks-ca\") pod \"81d0b3a1-0e93-491f-8a51-d071b7165936\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " Oct 30 03:01:35.470877 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.470785 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d0b3a1-0e93-491f-8a51-d071b7165936-kube-api-access-tskxq\" (UniqueName: \"kubernetes.io/projected/81d0b3a1-0e93-491f-8a51-d071b7165936-kube-api-access-tskxq\") pod \"81d0b3a1-0e93-491f-8a51-d071b7165936\" (UID: \"81d0b3a1-0e93-491f-8a51-d071b7165936\") " Oct 30 03:01:35.471168 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.471109 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-openshift-ca" pod "81d0b3a1-0e93-491f-8a51-d071b7165936" (UID: "81d0b3a1-0e93-491f-8a51-d071b7165936"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:01:35.471674 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.471511 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:01:35.475872 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.475751 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-xks-ca" pod "81d0b3a1-0e93-491f-8a51-d071b7165936" (UID: "81d0b3a1-0e93-491f-8a51-d071b7165936"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:01:35.476259 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.475780 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d0b3a1-0e93-491f-8a51-d071b7165936-kube-api-access-tskxq" pod "81d0b3a1-0e93-491f-8a51-d071b7165936" (UID: "81d0b3a1-0e93-491f-8a51-d071b7165936"). InnerVolumeSpecName "kube-api-access-tskxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:01:35.572577 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.572507 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81d0b3a1-0e93-491f-8a51-d071b7165936-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:01:35.572577 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:35.572569 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tskxq\" (UniqueName: \"kubernetes.io/projected/81d0b3a1-0e93-491f-8a51-d071b7165936-kube-api-access-tskxq\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:01:36.236922 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:36.236804 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" event={"ID":"81d0b3a1-0e93-491f-8a51-d071b7165936","Type":"ContainerDied","Data":"1de9b2d0e12739b28d3c02118132c28b1f45082307709b766da0359d32706421"} Oct 30 03:01:36.237646 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:36.236839 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv" Oct 30 03:01:36.237646 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:36.236989 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1de9b2d0e12739b28d3c02118132c28b1f45082307709b766da0359d32706421" Oct 30 03:01:45.530007 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:45.529945 5035 scope.go:122] "RemoveContainer" containerID="366006eba7b02e7b5b7829de0f2984582a721b1ddbfe6ba644e0b76967a420a8" Oct 30 03:01:47.467059 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.467011 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8"] Oct 30 03:01:47.467820 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.467793 5035 cpu_manager.go:414] "containerMap: removing container" podUID="81d0b3a1-0e93-491f-8a51-d071b7165936" containerName="revoke-keys" Oct 30 03:01:47.467898 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.467825 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d0b3a1-0e93-491f-8a51-d071b7165936" containerName="revoke-keys" Oct 30 03:01:47.467992 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.467977 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d0b3a1-0e93-491f-8a51-d071b7165936" containerName="revoke-keys" Oct 30 03:01:47.474481 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.474365 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.493341 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.493309 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-447d90-openshift-default-dockercfg-nnkmg\"" Oct 30 03:01:47.508470 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.508423 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8"] Oct 30 03:01:47.587486 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.587386 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-certs\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.588018 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.587638 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-data\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.588018 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.587749 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-socket\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.588018 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.587803 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63654084-c5cc-4af5-9837-f91d92b3276d-istiod-ca-cert\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.588018 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.587955 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63654084-c5cc-4af5-9837-f91d92b3276d-istio-podinfo\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.588249 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.588049 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-envoy\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.588249 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.588087 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-credential-socket\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.588249 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.588177 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-istio-token\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.588370 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.588350 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lg5\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-kube-api-access-t8lg5\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.670354 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:01:47.670304 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:01:47.689061 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.689011 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-certs\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.689061 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.689064 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-data\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.689370 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.689094 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-socket\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.689370 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.689122 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63654084-c5cc-4af5-9837-f91d92b3276d-istiod-ca-cert\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.689370 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.689153 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63654084-c5cc-4af5-9837-f91d92b3276d-istio-podinfo\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.689370 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.689180 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-envoy\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.689370 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.689206 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-credential-socket\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.689370 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.689253 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-istio-token\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.689766 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.689566 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-certs\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.690063 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.690038 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-data\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.690168 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.690046 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-credential-socket\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.690235 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.690178 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63654084-c5cc-4af5-9837-f91d92b3276d-istiod-ca-cert\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.690383 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.690362 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-socket\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.692021 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.690594 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lg5\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-kube-api-access-t8lg5\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.694440 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.692999 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-envoy\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.695503 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.695439 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63654084-c5cc-4af5-9837-f91d92b3276d-istio-podinfo\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.714612 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.714578 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-istio-token\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.714806 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.714763 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lg5\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-kube-api-access-t8lg5\") pod \"e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:47.793643 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:47.793516 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:48.117098 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:48.117067 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8"] Oct 30 03:01:48.119833 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:01:48.119803 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63654084_c5cc_4af5_9837_f91d92b3276d.slice/crio-9b0502af3f089403b0ec02107e979a6dd6fccaf1ee7547f4e2a478fd2e1f811b WatchSource:0}: Error finding container 9b0502af3f089403b0ec02107e979a6dd6fccaf1ee7547f4e2a478fd2e1f811b: Status 404 returned error can't find the container with id 9b0502af3f089403b0ec02107e979a6dd6fccaf1ee7547f4e2a478fd2e1f811b Oct 30 03:01:48.122225 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:48.122194 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:01:48.122303 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:48.122260 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:01:48.122303 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:48.122291 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:01:48.386069 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:48.385950 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" event={"ID":"63654084-c5cc-4af5-9837-f91d92b3276d","Type":"ContainerStarted","Data":"0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe"} Oct 30 03:01:48.386069 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:48.386025 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" event={"ID":"63654084-c5cc-4af5-9837-f91d92b3276d","Type":"ContainerStarted","Data":"9b0502af3f089403b0ec02107e979a6dd6fccaf1ee7547f4e2a478fd2e1f811b"} Oct 30 03:01:48.433936 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:48.433867 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" podStartSLOduration=1.433846838 podStartE2EDuration="1.433846838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:48.432090449 +0000 UTC m=+5586.935022670" watchObservedRunningTime="2026-10-30 03:01:48.433846838 +0000 UTC m=+5586.936779058" Oct 30 03:01:48.793638 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:48.793597 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:49.809062 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:49.808939 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:50.397456 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:50.397423 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:50.398317 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:50.398293 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:01:52.659087 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.659044 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv"] Oct 30 03:01:52.663333 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.663298 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:01:52.680672 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.680633 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-447d90\"" Oct 30 03:01:52.692830 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.692791 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8944a547-dockercfg-n2bld\"" Oct 30 03:01:52.697667 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.697632 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv"] Oct 30 03:01:52.749597 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.749520 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv\" (UID: \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:01:52.749802 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.749713 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnn8r\" (UniqueName: \"kubernetes.io/projected/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-kube-api-access-dnn8r\") pod \"payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv\" (UID: \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:01:52.850961 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.850920 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv\" (UID: \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:01:52.851170 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.850983 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnn8r\" (UniqueName: \"kubernetes.io/projected/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-kube-api-access-dnn8r\") pod \"payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv\" (UID: \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:01:52.851717 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.851692 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv\" (UID: \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:01:52.877185 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.877122 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnn8r\" (UniqueName: \"kubernetes.io/projected/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-kube-api-access-dnn8r\") pod \"payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv\" (UID: \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:01:52.978807 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:52.978668 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:01:53.309834 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:53.309798 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv"] Oct 30 03:01:53.311733 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:01:53.311704 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb327c5e_1ce7_4f8e_8bb3_cb6139f66795.slice/crio-65f69fe35462ddc7fbd0055dec72cca8feb62cfbf973c1bb6497d22b9749a704 WatchSource:0}: Error finding container 65f69fe35462ddc7fbd0055dec72cca8feb62cfbf973c1bb6497d22b9749a704: Status 404 returned error can't find the container with id 65f69fe35462ddc7fbd0055dec72cca8feb62cfbf973c1bb6497d22b9749a704 Oct 30 03:01:53.415723 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:53.415678 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" event={"ID":"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795","Type":"ContainerStarted","Data":"a1ea7497136b6aeea8d4c4633dee238c4b759dc504a7f8071d158559598ee810"} Oct 30 03:01:53.415945 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:53.415736 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" event={"ID":"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795","Type":"ContainerStarted","Data":"65f69fe35462ddc7fbd0055dec72cca8feb62cfbf973c1bb6497d22b9749a704"} Oct 30 03:01:53.415945 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:01:53.415919 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:02:04.429134 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:04.429100 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:02:04.471223 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:04.471131 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" podStartSLOduration=12.471113668 podStartE2EDuration="12.471113668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:53.457193859 +0000 UTC m=+5591.960126092" watchObservedRunningTime="2026-10-30 03:02:04.471113668 +0000 UTC m=+5602.974045946" Oct 30 03:02:06.404429 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:06.401773 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:02:06.404429 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:06.402006 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:02:06.517638 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:06.517501 5035 generic.go:356] "Generic (PLEG): container finished" podID="63654084-c5cc-4af5-9837-f91d92b3276d" containerID="0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe" exitCode=0 Oct 30 03:02:06.517847 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:06.517725 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" event={"ID":"63654084-c5cc-4af5-9837-f91d92b3276d","Type":"ContainerDied","Data":"0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe"} Oct 30 03:02:06.518204 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:06.518184 5035 scope.go:122] "RemoveContainer" containerID="0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe" Oct 30 03:02:06.519625 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:06.519592 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:02:06.519723 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:06.519669 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:02:06.519723 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:06.519703 5035 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:02:07.551902 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.551767 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" event={"ID":"63654084-c5cc-4af5-9837-f91d92b3276d","Type":"ContainerStarted","Data":"dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7"} Oct 30 03:02:07.794303 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.794258 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:02:07.794680 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.794649 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 03:02:07.794832 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.794705 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 03:02:07.905978 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.905874 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j"] Oct 30 03:02:07.908946 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.908920 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:07.924044 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.924010 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-447d90\"/\"default-dockercfg-rgqw2\"" Oct 30 03:02:07.924608 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.924586 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-447d90\"/\"kube-root-ca.crt\"" Oct 30 03:02:07.924725 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.924654 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-447d90\"/\"openshift-service-ca.crt\"" Oct 30 03:02:07.925066 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.925047 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-447d90\"/\"simulator-single-main-447d90-kserve-self-signed-certs\"" Oct 30 03:02:07.933298 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:07.933267 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j"] Oct 30 03:02:08.042792 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.042746 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-home\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.042792 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.042791 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-dshm\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.043065 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.042816 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-tmp-dir\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.043065 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.042977 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-model-cache\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.043065 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.043033 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rct\" (UniqueName: \"kubernetes.io/projected/4b793f8b-7d28-423f-9433-1eb283f1263d-kube-api-access-22rct\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.043245 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.043151 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b793f8b-7d28-423f-9433-1eb283f1263d-tls-certs\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.144080 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.144027 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-home\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.144080 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.144083 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-dshm\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.144379 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.144110 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-tmp-dir\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.144379 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.144274 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-model-cache\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.144379 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.144342 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rct\" (UniqueName: \"kubernetes.io/projected/4b793f8b-7d28-423f-9433-1eb283f1263d-kube-api-access-22rct\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.144591 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.144383 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b793f8b-7d28-423f-9433-1eb283f1263d-tls-certs\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.144652 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.144580 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-home\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.144652 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.144638 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-tmp-dir\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.144784 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.144762 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-model-cache\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.146651 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.146622 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-dshm\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.146940 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.146918 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b793f8b-7d28-423f-9433-1eb283f1263d-tls-certs\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.176599 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.176173 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rct\" (UniqueName: \"kubernetes.io/projected/4b793f8b-7d28-423f-9433-1eb283f1263d-kube-api-access-22rct\") pod \"simulator-single-main-447d90-kserve-c77d44d8-rzv2j\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.222498 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.222383 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:08.604094 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.604057 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j"] Oct 30 03:02:08.606188 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:02:08.606145 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b793f8b_7d28_423f_9433_1eb283f1263d.slice/crio-a8417a68e96ce84409d2ebde4c3f05713b8653ecd96e6982dd14bfd42a8f7689 WatchSource:0}: Error finding container a8417a68e96ce84409d2ebde4c3f05713b8653ecd96e6982dd14bfd42a8f7689: Status 404 returned error can't find the container with id a8417a68e96ce84409d2ebde4c3f05713b8653ecd96e6982dd14bfd42a8f7689 Oct 30 03:02:08.794014 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.793968 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 03:02:08.794225 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:08.794046 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 03:02:09.593099 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:09.593056 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" event={"ID":"4b793f8b-7d28-423f-9433-1eb283f1263d","Type":"ContainerStarted","Data":"57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce"} Oct 30 03:02:09.593312 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:09.593110 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" event={"ID":"4b793f8b-7d28-423f-9433-1eb283f1263d","Type":"ContainerStarted","Data":"a8417a68e96ce84409d2ebde4c3f05713b8653ecd96e6982dd14bfd42a8f7689"} Oct 30 03:02:09.637033 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:09.636963 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" podStartSLOduration=2.636947418 podStartE2EDuration="2.636947418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:09.634473415 +0000 UTC m=+5608.137405634" watchObservedRunningTime="2026-10-30 03:02:09.636947418 +0000 UTC m=+5608.139879637" Oct 30 03:02:09.794094 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:09.794044 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 03:02:09.794330 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:09.794106 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 03:02:10.222791 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:10.222729 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:10.794872 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:10.794790 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 03:02:10.795743 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:10.794913 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 03:02:11.793961 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:11.793917 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 03:02:11.794192 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:11.793978 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 03:02:12.794048 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:12.793990 5035 patch_prober.go:29] interesting pod/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Oct 30 03:02:12.794795 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:12.794081 5035 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Oct 30 03:02:13.798587 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:13.798545 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:02:13.799011 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:13.798969 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:02:13.799896 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:13.799870 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:02:17.633236 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:02:17.633190 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:02:18.223428 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:18.223387 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:18.236493 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:18.236460 5035 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:18.670947 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:18.670855 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:02:27.044859 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:27.044762 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh"] Oct 30 03:02:27.056022 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:27.055978 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5trjh"] Oct 30 03:02:27.642450 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:27.642406 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d4574d-d573-46dd-bea0-c139be3fa2e3" path="/var/lib/kubelet/pods/e3d4574d-d573-46dd-bea0-c139be3fa2e3/volumes" Oct 30 03:02:40.045182 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.045111 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4"] Oct 30 03:02:40.048866 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.048830 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.077574 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.077516 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:02:40.077762 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.077516 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:02:40.077762 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.077641 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:02:40.078141 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.078106 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:02:40.081128 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.081091 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4"] Oct 30 03:02:40.158366 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.158321 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9727a5d-jvbw4\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.158628 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.158463 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljg95\" (UniqueName: \"kubernetes.io/projected/e626c0b2-7c2a-45ab-99d2-9463524e4408-kube-api-access-ljg95\") pod \"maas-api-revoke-sub-c9727a5d-jvbw4\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.158628 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.158510 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9727a5d-jvbw4\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.259462 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.259422 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9727a5d-jvbw4\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.259686 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.259504 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljg95\" (UniqueName: \"kubernetes.io/projected/e626c0b2-7c2a-45ab-99d2-9463524e4408-kube-api-access-ljg95\") pod \"maas-api-revoke-sub-c9727a5d-jvbw4\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.259686 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.259588 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9727a5d-jvbw4\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.260320 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.260296 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9727a5d-jvbw4\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.262132 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.262099 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9727a5d-jvbw4\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.283498 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.283453 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljg95\" (UniqueName: \"kubernetes.io/projected/e626c0b2-7c2a-45ab-99d2-9463524e4408-kube-api-access-ljg95\") pod \"maas-api-revoke-sub-c9727a5d-jvbw4\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.363010 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.362919 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:40.677652 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.677521 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4"] Oct 30 03:02:40.682482 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:02:40.682375 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode626c0b2_7c2a_45ab_99d2_9463524e4408.slice/crio-326261f51d6f08d8261b83e2ec1db7fd101531ef3c084cee2b56bfd15038791f WatchSource:0}: Error finding container 326261f51d6f08d8261b83e2ec1db7fd101531ef3c084cee2b56bfd15038791f: Status 404 returned error can't find the container with id 326261f51d6f08d8261b83e2ec1db7fd101531ef3c084cee2b56bfd15038791f Oct 30 03:02:40.811786 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:40.811748 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" event={"ID":"e626c0b2-7c2a-45ab-99d2-9463524e4408","Type":"ContainerStarted","Data":"326261f51d6f08d8261b83e2ec1db7fd101531ef3c084cee2b56bfd15038791f"} Oct 30 03:02:41.820613 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:41.820490 5035 generic.go:356] "Generic (PLEG): container finished" podID="e626c0b2-7c2a-45ab-99d2-9463524e4408" containerID="609b5582b3e06d2db590e2ef3a38b367680597be83c5356a1df2c5e8f61041bc" exitCode=0 Oct 30 03:02:41.820613 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:41.820578 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" event={"ID":"e626c0b2-7c2a-45ab-99d2-9463524e4408","Type":"ContainerDied","Data":"609b5582b3e06d2db590e2ef3a38b367680597be83c5356a1df2c5e8f61041bc"} Oct 30 03:02:43.004833 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.004800 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:43.200731 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.200675 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e626c0b2-7c2a-45ab-99d2-9463524e4408-kube-api-access-ljg95\" (UniqueName: \"kubernetes.io/projected/e626c0b2-7c2a-45ab-99d2-9463524e4408-kube-api-access-ljg95\") pod \"e626c0b2-7c2a-45ab-99d2-9463524e4408\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " Oct 30 03:02:43.200962 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.200835 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-openshift-ca\") pod \"e626c0b2-7c2a-45ab-99d2-9463524e4408\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " Oct 30 03:02:43.201022 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.200970 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-xks-ca\") pod \"e626c0b2-7c2a-45ab-99d2-9463524e4408\" (UID: \"e626c0b2-7c2a-45ab-99d2-9463524e4408\") " Oct 30 03:02:43.201351 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.201303 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-openshift-ca" pod "e626c0b2-7c2a-45ab-99d2-9463524e4408" (UID: "e626c0b2-7c2a-45ab-99d2-9463524e4408"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:02:43.201477 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.201459 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:02:43.203515 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.203472 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-xks-ca" pod "e626c0b2-7c2a-45ab-99d2-9463524e4408" (UID: "e626c0b2-7c2a-45ab-99d2-9463524e4408"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:02:43.203695 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.203587 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e626c0b2-7c2a-45ab-99d2-9463524e4408-kube-api-access-ljg95" pod "e626c0b2-7c2a-45ab-99d2-9463524e4408" (UID: "e626c0b2-7c2a-45ab-99d2-9463524e4408"). InnerVolumeSpecName "kube-api-access-ljg95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:02:43.302821 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.302774 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626c0b2-7c2a-45ab-99d2-9463524e4408-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:02:43.302821 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.302817 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljg95\" (UniqueName: \"kubernetes.io/projected/e626c0b2-7c2a-45ab-99d2-9463524e4408-kube-api-access-ljg95\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:02:43.840504 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.840470 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" Oct 30 03:02:43.840738 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.840508 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4" event={"ID":"e626c0b2-7c2a-45ab-99d2-9463524e4408","Type":"ContainerDied","Data":"326261f51d6f08d8261b83e2ec1db7fd101531ef3c084cee2b56bfd15038791f"} Oct 30 03:02:43.840738 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:43.840589 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="326261f51d6f08d8261b83e2ec1db7fd101531ef3c084cee2b56bfd15038791f" Oct 30 03:02:47.630067 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:02:47.630020 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:02:50.045399 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:50.045360 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms"] Oct 30 03:02:50.056623 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:50.056588 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-665ms"] Oct 30 03:02:51.642443 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:51.642396 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb" path="/var/lib/kubelet/pods/cd2cafdd-d7e8-4475-a5d7-58c4b1006fdb/volumes" Oct 30 03:02:56.078240 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:56.078188 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j"] Oct 30 03:02:57.380798 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:02:57.380754 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv"] Oct 30 03:03:02.381360 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:02.381293 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" podUID="fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" containerName="payload-processing" containerID="cri-o://a1ea7497136b6aeea8d4c4633dee238c4b759dc504a7f8071d158559598ee810" gracePeriod=25 Oct 30 03:03:04.425512 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:04.425441 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" podUID="fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:03:07.488996 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:07.488952 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8"] Oct 30 03:03:07.489842 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:07.489808 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" containerID="cri-o://dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7" gracePeriod=30 Oct 30 03:03:08.773997 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.773945 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv"] Oct 30 03:03:08.827601 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.827558 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l4jjv"] Oct 30 03:03:08.887182 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.887133 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh"] Oct 30 03:03:08.887720 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.887705 5035 cpu_manager.go:414] "containerMap: removing container" podUID="e626c0b2-7c2a-45ab-99d2-9463524e4408" containerName="revoke-keys" Oct 30 03:03:08.887777 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.887721 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e626c0b2-7c2a-45ab-99d2-9463524e4408" containerName="revoke-keys" Oct 30 03:03:08.887832 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.887822 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="e626c0b2-7c2a-45ab-99d2-9463524e4408" containerName="revoke-keys" Oct 30 03:03:08.890896 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.890859 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:08.908642 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.908596 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:03:08.920518 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.920479 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:03:08.920518 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.920547 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:03:08.925363 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:08.925313 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh"] Oct 30 03:03:09.007215 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.007164 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:03:09.016071 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.016012 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wtrvh\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.016330 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.016153 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wtrvh\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.016330 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.016210 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9k6\" (UniqueName: \"kubernetes.io/projected/d9526682-8fe9-490f-a151-5e220f1eac4e-kube-api-access-gm9k6\") pod \"maas-api-revoke-sub-3d4156f1-wtrvh\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.117761 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.117662 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wtrvh\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.117761 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.117730 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wtrvh\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.117977 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.117775 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9k6\" (UniqueName: \"kubernetes.io/projected/d9526682-8fe9-490f-a151-5e220f1eac4e-kube-api-access-gm9k6\") pod \"maas-api-revoke-sub-3d4156f1-wtrvh\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.118471 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.118445 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wtrvh\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.120213 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.120188 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wtrvh\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.144283 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.144191 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9k6\" (UniqueName: \"kubernetes.io/projected/d9526682-8fe9-490f-a151-5e220f1eac4e-kube-api-access-gm9k6\") pod \"maas-api-revoke-sub-3d4156f1-wtrvh\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.207871 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.207820 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:09.520376 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.520347 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh"] Oct 30 03:03:09.522042 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:03:09.522014 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9526682_8fe9_490f_a151_5e220f1eac4e.slice/crio-e34c1e785690d3e911228356b5880c703871bd58248b0b235f750bcae71fea48 WatchSource:0}: Error finding container e34c1e785690d3e911228356b5880c703871bd58248b0b235f750bcae71fea48: Status 404 returned error can't find the container with id e34c1e785690d3e911228356b5880c703871bd58248b0b235f750bcae71fea48 Oct 30 03:03:09.642704 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:09.642666 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d0b3a1-0e93-491f-8a51-d071b7165936" path="/var/lib/kubelet/pods/81d0b3a1-0e93-491f-8a51-d071b7165936/volumes" Oct 30 03:03:10.020891 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:10.020846 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" event={"ID":"d9526682-8fe9-490f-a151-5e220f1eac4e","Type":"ContainerStarted","Data":"1828069aa4fde22e5ca869ab66f4f9ccb2adf411cf5d63ded6e1d9c1da3d6184"} Oct 30 03:03:10.021417 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:10.020912 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" event={"ID":"d9526682-8fe9-490f-a151-5e220f1eac4e","Type":"ContainerStarted","Data":"e34c1e785690d3e911228356b5880c703871bd58248b0b235f750bcae71fea48"} Oct 30 03:03:10.044980 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:10.044894 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr"] Oct 30 03:03:10.050100 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:10.050072 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2crjr"] Oct 30 03:03:10.067890 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:10.067820 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" podStartSLOduration=2.067798805 podStartE2EDuration="2.067798805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:10.065977149 +0000 UTC m=+5668.568909371" watchObservedRunningTime="2026-10-30 03:03:10.067798805 +0000 UTC m=+5668.570731025" Oct 30 03:03:11.028072 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.028033 5035 generic.go:356] "Generic (PLEG): container finished" podID="d9526682-8fe9-490f-a151-5e220f1eac4e" containerID="1828069aa4fde22e5ca869ab66f4f9ccb2adf411cf5d63ded6e1d9c1da3d6184" exitCode=0 Oct 30 03:03:11.028509 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.028116 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" event={"ID":"d9526682-8fe9-490f-a151-5e220f1eac4e","Type":"ContainerDied","Data":"1828069aa4fde22e5ca869ab66f4f9ccb2adf411cf5d63ded6e1d9c1da3d6184"} Oct 30 03:03:11.100607 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.100553 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" podUID="4b793f8b-7d28-423f-9433-1eb283f1263d" containerName="main" containerID="cri-o://57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce" gracePeriod=45 Oct 30 03:03:11.376274 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.376245 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:03:11.446060 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446007 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-model-cache\") pod \"4b793f8b-7d28-423f-9433-1eb283f1263d\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " Oct 30 03:03:11.446255 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446161 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-home\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-home\") pod \"4b793f8b-7d28-423f-9433-1eb283f1263d\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " Oct 30 03:03:11.446255 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446214 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-dshm\") pod \"4b793f8b-7d28-423f-9433-1eb283f1263d\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " Oct 30 03:03:11.446382 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446289 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b793f8b-7d28-423f-9433-1eb283f1263d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b793f8b-7d28-423f-9433-1eb283f1263d-tls-certs\") pod \"4b793f8b-7d28-423f-9433-1eb283f1263d\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " Oct 30 03:03:11.446382 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446319 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-tmp-dir\") pod \"4b793f8b-7d28-423f-9433-1eb283f1263d\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " Oct 30 03:03:11.446500 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446369 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-model-cache" pod "4b793f8b-7d28-423f-9433-1eb283f1263d" (UID: "4b793f8b-7d28-423f-9433-1eb283f1263d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:11.446500 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446417 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b793f8b-7d28-423f-9433-1eb283f1263d-kube-api-access-22rct\" (UniqueName: \"kubernetes.io/projected/4b793f8b-7d28-423f-9433-1eb283f1263d-kube-api-access-22rct\") pod \"4b793f8b-7d28-423f-9433-1eb283f1263d\" (UID: \"4b793f8b-7d28-423f-9433-1eb283f1263d\") " Oct 30 03:03:11.446667 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446491 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-home" pod "4b793f8b-7d28-423f-9433-1eb283f1263d" (UID: "4b793f8b-7d28-423f-9433-1eb283f1263d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:11.446732 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446713 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-tmp-dir" pod "4b793f8b-7d28-423f-9433-1eb283f1263d" (UID: "4b793f8b-7d28-423f-9433-1eb283f1263d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:11.446864 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446844 5035 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-tmp-dir\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:11.446936 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446870 5035 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-model-cache\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:11.446936 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.446884 5035 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-home\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:11.448950 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.448916 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b793f8b-7d28-423f-9433-1eb283f1263d-kube-api-access-22rct" pod "4b793f8b-7d28-423f-9433-1eb283f1263d" (UID: "4b793f8b-7d28-423f-9433-1eb283f1263d"). InnerVolumeSpecName "kube-api-access-22rct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:03:11.449593 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.449564 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-dshm" pod "4b793f8b-7d28-423f-9433-1eb283f1263d" (UID: "4b793f8b-7d28-423f-9433-1eb283f1263d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:11.449593 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.449581 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b793f8b-7d28-423f-9433-1eb283f1263d-tls-certs" pod "4b793f8b-7d28-423f-9433-1eb283f1263d" (UID: "4b793f8b-7d28-423f-9433-1eb283f1263d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:03:11.547421 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.547382 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22rct\" (UniqueName: \"kubernetes.io/projected/4b793f8b-7d28-423f-9433-1eb283f1263d-kube-api-access-22rct\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:11.547421 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.547416 5035 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b793f8b-7d28-423f-9433-1eb283f1263d-dshm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:11.547421 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.547426 5035 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b793f8b-7d28-423f-9433-1eb283f1263d-tls-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:11.642497 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:11.642413 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710325c1-2556-4e53-b0be-e71662734b17" path="/var/lib/kubelet/pods/710325c1-2556-4e53-b0be-e71662734b17/volumes" Oct 30 03:03:12.036604 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.036557 5035 generic.go:356] "Generic (PLEG): container finished" podID="4b793f8b-7d28-423f-9433-1eb283f1263d" containerID="57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce" exitCode=0 Oct 30 03:03:12.036604 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.036592 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" Oct 30 03:03:12.037228 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.036632 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" event={"ID":"4b793f8b-7d28-423f-9433-1eb283f1263d","Type":"ContainerDied","Data":"57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce"} Oct 30 03:03:12.037228 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.036681 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j" event={"ID":"4b793f8b-7d28-423f-9433-1eb283f1263d","Type":"ContainerDied","Data":"a8417a68e96ce84409d2ebde4c3f05713b8653ecd96e6982dd14bfd42a8f7689"} Oct 30 03:03:12.037228 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.036702 5035 scope.go:122] "RemoveContainer" containerID="57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce" Oct 30 03:03:12.049635 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.049597 5035 scope.go:122] "RemoveContainer" containerID="57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce" Oct 30 03:03:12.050189 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:03:12.050153 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce\": container with ID starting with 57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce not found: ID does not exist" containerID="57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce" Oct 30 03:03:12.050293 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.050209 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce"} err="failed to get container status \"57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce\": rpc error: code = NotFound desc = could not find container \"57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce\": container with ID starting with 57245452d9cbc66082b5c4aab8e0abcdcba969a45cf99231d696205f223f6fce not found: ID does not exist" Oct 30 03:03:12.090605 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.090558 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j"] Oct 30 03:03:12.102917 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.102862 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-447d90/simulator-single-main-447d90-kserve-c77d44d8-rzv2j"] Oct 30 03:03:12.203304 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.203274 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:12.257611 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.257573 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9526682-8fe9-490f-a151-5e220f1eac4e-kube-api-access-gm9k6\" (UniqueName: \"kubernetes.io/projected/d9526682-8fe9-490f-a151-5e220f1eac4e-kube-api-access-gm9k6\") pod \"d9526682-8fe9-490f-a151-5e220f1eac4e\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " Oct 30 03:03:12.257834 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.257638 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-openshift-ca\") pod \"d9526682-8fe9-490f-a151-5e220f1eac4e\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " Oct 30 03:03:12.257834 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.257737 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-xks-ca\") pod \"d9526682-8fe9-490f-a151-5e220f1eac4e\" (UID: \"d9526682-8fe9-490f-a151-5e220f1eac4e\") " Oct 30 03:03:12.258797 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.258752 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-openshift-ca" pod "d9526682-8fe9-490f-a151-5e220f1eac4e" (UID: "d9526682-8fe9-490f-a151-5e220f1eac4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:03:12.260679 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.260523 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9526682-8fe9-490f-a151-5e220f1eac4e-kube-api-access-gm9k6" pod "d9526682-8fe9-490f-a151-5e220f1eac4e" (UID: "d9526682-8fe9-490f-a151-5e220f1eac4e"). InnerVolumeSpecName "kube-api-access-gm9k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:03:12.260679 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.260627 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-xks-ca" pod "d9526682-8fe9-490f-a151-5e220f1eac4e" (UID: "d9526682-8fe9-490f-a151-5e220f1eac4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:03:12.358952 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.358922 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.358952 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.358954 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm9k6\" (UniqueName: \"kubernetes.io/projected/d9526682-8fe9-490f-a151-5e220f1eac4e-kube-api-access-gm9k6\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.359126 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.358964 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9526682-8fe9-490f-a151-5e220f1eac4e-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.805311 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.805272 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:03:12.866278 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866177 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-socket\") pod \"63654084-c5cc-4af5-9837-f91d92b3276d\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " Oct 30 03:03:12.866278 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866249 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-kube-api-access-t8lg5\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-kube-api-access-t8lg5\") pod \"63654084-c5cc-4af5-9837-f91d92b3276d\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " Oct 30 03:03:12.866618 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866307 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63654084-c5cc-4af5-9837-f91d92b3276d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63654084-c5cc-4af5-9837-f91d92b3276d-istiod-ca-cert\") pod \"63654084-c5cc-4af5-9837-f91d92b3276d\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " Oct 30 03:03:12.866618 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866367 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-certs\") pod \"63654084-c5cc-4af5-9837-f91d92b3276d\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " Oct 30 03:03:12.866618 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866399 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-istio-token\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-istio-token\") pod \"63654084-c5cc-4af5-9837-f91d92b3276d\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " Oct 30 03:03:12.866618 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866471 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-credential-socket\") pod \"63654084-c5cc-4af5-9837-f91d92b3276d\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " Oct 30 03:03:12.866941 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866585 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-socket" pod "63654084-c5cc-4af5-9837-f91d92b3276d" (UID: "63654084-c5cc-4af5-9837-f91d92b3276d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:12.866941 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866595 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-envoy\") pod \"63654084-c5cc-4af5-9837-f91d92b3276d\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " Oct 30 03:03:12.866941 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866771 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63654084-c5cc-4af5-9837-f91d92b3276d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63654084-c5cc-4af5-9837-f91d92b3276d-istio-podinfo\") pod \"63654084-c5cc-4af5-9837-f91d92b3276d\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " Oct 30 03:03:12.866941 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866811 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-data\") pod \"63654084-c5cc-4af5-9837-f91d92b3276d\" (UID: \"63654084-c5cc-4af5-9837-f91d92b3276d\") " Oct 30 03:03:12.866941 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.866867 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-certs" pod "63654084-c5cc-4af5-9837-f91d92b3276d" (UID: "63654084-c5cc-4af5-9837-f91d92b3276d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:12.867229 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.867005 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-credential-socket" pod "63654084-c5cc-4af5-9837-f91d92b3276d" (UID: "63654084-c5cc-4af5-9837-f91d92b3276d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:12.867229 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.867023 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63654084-c5cc-4af5-9837-f91d92b3276d-istiod-ca-cert" pod "63654084-c5cc-4af5-9837-f91d92b3276d" (UID: "63654084-c5cc-4af5-9837-f91d92b3276d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:03:12.867332 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.867273 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-data" pod "63654084-c5cc-4af5-9837-f91d92b3276d" (UID: "63654084-c5cc-4af5-9837-f91d92b3276d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:12.867615 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.867501 5035 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-data\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.867615 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.867544 5035 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.867615 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.867560 5035 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63654084-c5cc-4af5-9837-f91d92b3276d-istiod-ca-cert\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.867615 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.867576 5035 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-workload-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.867615 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.867589 5035 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-credential-socket\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.869564 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.869503 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63654084-c5cc-4af5-9837-f91d92b3276d-istio-podinfo" pod "63654084-c5cc-4af5-9837-f91d92b3276d" (UID: "63654084-c5cc-4af5-9837-f91d92b3276d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:03:12.869765 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.869740 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-envoy" pod "63654084-c5cc-4af5-9837-f91d92b3276d" (UID: "63654084-c5cc-4af5-9837-f91d92b3276d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:12.869839 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.869821 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-kube-api-access-t8lg5" pod "63654084-c5cc-4af5-9837-f91d92b3276d" (UID: "63654084-c5cc-4af5-9837-f91d92b3276d"). InnerVolumeSpecName "kube-api-access-t8lg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:03:12.869884 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.869853 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-istio-token" pod "63654084-c5cc-4af5-9837-f91d92b3276d" (UID: "63654084-c5cc-4af5-9837-f91d92b3276d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:03:12.969120 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.969072 5035 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63654084-c5cc-4af5-9837-f91d92b3276d-istio-podinfo\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.969120 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.969114 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8lg5\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-kube-api-access-t8lg5\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.969120 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.969125 5035 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63654084-c5cc-4af5-9837-f91d92b3276d-istio-token\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:12.969120 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:12.969136 5035 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63654084-c5cc-4af5-9837-f91d92b3276d-istio-envoy\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:13.053935 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.053901 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" Oct 30 03:03:13.054458 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.053903 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh" event={"ID":"d9526682-8fe9-490f-a151-5e220f1eac4e","Type":"ContainerDied","Data":"e34c1e785690d3e911228356b5880c703871bd58248b0b235f750bcae71fea48"} Oct 30 03:03:13.054458 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.054057 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e34c1e785690d3e911228356b5880c703871bd58248b0b235f750bcae71fea48" Oct 30 03:03:13.055870 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.055842 5035 generic.go:356] "Generic (PLEG): container finished" podID="63654084-c5cc-4af5-9837-f91d92b3276d" containerID="dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7" exitCode=0 Oct 30 03:03:13.056019 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.055899 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" Oct 30 03:03:13.056019 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.055931 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" event={"ID":"63654084-c5cc-4af5-9837-f91d92b3276d","Type":"ContainerDied","Data":"dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7"} Oct 30 03:03:13.056019 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.055989 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8" event={"ID":"63654084-c5cc-4af5-9837-f91d92b3276d","Type":"ContainerDied","Data":"9b0502af3f089403b0ec02107e979a6dd6fccaf1ee7547f4e2a478fd2e1f811b"} Oct 30 03:03:13.056019 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.056019 5035 scope.go:122] "RemoveContainer" containerID="dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7" Oct 30 03:03:13.066544 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.066499 5035 scope.go:122] "RemoveContainer" containerID="0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe" Oct 30 03:03:13.076241 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.076215 5035 scope.go:122] "RemoveContainer" containerID="dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7" Oct 30 03:03:13.076578 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:03:13.076555 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7\": container with ID starting with dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7 not found: ID does not exist" containerID="dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7" Oct 30 03:03:13.076642 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.076590 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7"} err="failed to get container status \"dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7\": rpc error: code = NotFound desc = could not find container \"dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7\": container with ID starting with dff7a78590aa4c994b38400a492c12c181eecbf35813f3599263d30adc4d18c7 not found: ID does not exist" Oct 30 03:03:13.076642 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.076609 5035 scope.go:122] "RemoveContainer" containerID="0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe" Oct 30 03:03:13.076854 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:03:13.076839 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe\": container with ID starting with 0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe not found: ID does not exist" containerID="0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe" Oct 30 03:03:13.076900 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.076859 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe"} err="failed to get container status \"0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe\": rpc error: code = NotFound desc = could not find container \"0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe\": container with ID starting with 0fedfcd08be20234d3b51f3befed04328475b1020667bee61d184901c48d3bfe not found: ID does not exist" Oct 30 03:03:13.115372 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.115316 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8"] Oct 30 03:03:13.120579 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.120472 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-447d90-openshift-default-6c9688cb89swbk8"] Oct 30 03:03:13.641889 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.641848 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b793f8b-7d28-423f-9433-1eb283f1263d" path="/var/lib/kubelet/pods/4b793f8b-7d28-423f-9433-1eb283f1263d/volumes" Oct 30 03:03:13.642458 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:13.642436 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" path="/var/lib/kubelet/pods/63654084-c5cc-4af5-9837-f91d92b3276d/volumes" Oct 30 03:03:14.424646 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:14.424599 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" podUID="fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:03:17.631345 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:03:17.631298 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:03:22.038553 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:22.037589 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv"] Oct 30 03:03:22.073572 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:22.067305 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mfbdv"] Oct 30 03:03:23.295867 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.295729 5035 generic.go:356] "Generic (PLEG): container finished" podID="fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" containerID="a1ea7497136b6aeea8d4c4633dee238c4b759dc504a7f8071d158559598ee810" exitCode=0 Oct 30 03:03:23.296763 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.296700 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" event={"ID":"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795","Type":"ContainerDied","Data":"a1ea7497136b6aeea8d4c4633dee238c4b759dc504a7f8071d158559598ee810"} Oct 30 03:03:23.312563 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.312478 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:03:23.381580 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.371983 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-plugins-config-volume\") pod \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\" (UID: \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\") " Oct 30 03:03:23.406839 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.406798 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-kube-api-access-dnn8r\" (UniqueName: \"kubernetes.io/projected/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-kube-api-access-dnn8r\") pod \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\" (UID: \"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795\") " Oct 30 03:03:23.431904 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.431607 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-plugins-config-volume" pod "fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" (UID: "fb327c5e-1ce7-4f8e-8bb3-cb6139f66795"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:03:23.467070 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.466741 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-kube-api-access-dnn8r" pod "fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" (UID: "fb327c5e-1ce7-4f8e-8bb3-cb6139f66795"). InnerVolumeSpecName "kube-api-access-dnn8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:03:23.467700 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.467636 5035 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-plugins-config-volume\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:23.467700 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.467670 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnn8r\" (UniqueName: \"kubernetes.io/projected/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795-kube-api-access-dnn8r\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:03:23.649063 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:23.648945 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3622d2-10ba-48e8-8447-ffba317ab42e" path="/var/lib/kubelet/pods/bf3622d2-10ba-48e8-8447-ffba317ab42e/volumes" Oct 30 03:03:24.302844 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:24.302807 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" Oct 30 03:03:24.303372 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:24.302811 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv" event={"ID":"fb327c5e-1ce7-4f8e-8bb3-cb6139f66795","Type":"ContainerDied","Data":"65f69fe35462ddc7fbd0055dec72cca8feb62cfbf973c1bb6497d22b9749a704"} Oct 30 03:03:24.303372 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:24.302956 5035 scope.go:122] "RemoveContainer" containerID="a1ea7497136b6aeea8d4c4633dee238c4b759dc504a7f8071d158559598ee810" Oct 30 03:03:24.355873 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:24.355836 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv"] Oct 30 03:03:24.368813 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:24.368762 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-447d90-9f6c86d76tnrjv"] Oct 30 03:03:25.649240 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:25.649150 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" path="/var/lib/kubelet/pods/fb327c5e-1ce7-4f8e-8bb3-cb6139f66795/volumes" Oct 30 03:03:44.268061 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:44.268024 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 03:03:44.280027 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:44.279995 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 03:03:44.285224 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:44.285191 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 03:03:44.391524 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:44.391491 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 03:03:44.402733 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:44.402702 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 03:03:44.407376 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:44.407351 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 03:03:45.679652 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:45.679612 5035 scope.go:122] "RemoveContainer" containerID="3da2f247a0091affbf7c2615138e2d47641406f85d0b67b01a0f4158271d0a42" Oct 30 03:03:47.629160 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:03:47.629112 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:03:50.036160 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:50.036120 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t"] Oct 30 03:03:50.047844 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:50.047804 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4xn5t"] Oct 30 03:03:51.641420 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:03:51.641376 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebfbff74-8164-48c3-9dcb-76b9f3198923" path="/var/lib/kubelet/pods/ebfbff74-8164-48c3-9dcb-76b9f3198923/volumes" Oct 30 03:04:00.032076 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:00.031987 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt"] Oct 30 03:04:00.043776 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:00.043738 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-62cwt"] Oct 30 03:04:01.642587 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:01.642522 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d839b5d-c221-4e6a-9bd4-8bda77c68431" path="/var/lib/kubelet/pods/5d839b5d-c221-4e6a-9bd4-8bda77c68431/volumes" Oct 30 03:04:17.630571 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:04:17.630504 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:04:21.033186 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:21.033149 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr"] Oct 30 03:04:21.045081 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:21.045044 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-qqxcr"] Oct 30 03:04:21.642869 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:21.642829 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f0a2df-aef0-4a8a-8465-959dc18036c8" path="/var/lib/kubelet/pods/b2f0a2df-aef0-4a8a-8465-959dc18036c8/volumes" Oct 30 03:04:39.036865 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:39.036819 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g"] Oct 30 03:04:39.048773 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:39.048733 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-57g6g"] Oct 30 03:04:39.642803 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:39.642766 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="374f9c27-0f52-4f56-abde-233714fa6e04" path="/var/lib/kubelet/pods/374f9c27-0f52-4f56-abde-233714fa6e04/volumes" Oct 30 03:04:45.722113 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:45.722075 5035 scope.go:122] "RemoveContainer" containerID="e03a79a0ad8933477672f560987edd3fb2ee5f43ab1eaee81880d6ac57b59fb1" Oct 30 03:04:45.731813 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:45.731787 5035 scope.go:122] "RemoveContainer" containerID="f11cfb957eb8506985b7fb5f988ee9ba438cee3e95721ef332b1e81ec31bf6f0" Oct 30 03:04:45.741431 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:45.741404 5035 scope.go:122] "RemoveContainer" containerID="914bcafcba8a125e37ba021fb3a360bcd4a7559e63630a4d56a1b72971122035" Oct 30 03:04:47.650274 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:04:47.650229 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:04:54.025108 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025059 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w"] Oct 30 03:04:54.025593 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025577 5035 cpu_manager.go:414] "containerMap: removing container" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" Oct 30 03:04:54.025645 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025595 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" Oct 30 03:04:54.025645 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025614 5035 cpu_manager.go:414] "containerMap: removing container" podUID="d9526682-8fe9-490f-a151-5e220f1eac4e" containerName="revoke-keys" Oct 30 03:04:54.025645 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025619 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9526682-8fe9-490f-a151-5e220f1eac4e" containerName="revoke-keys" Oct 30 03:04:54.025752 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025653 5035 cpu_manager.go:414] "containerMap: removing container" podUID="4b793f8b-7d28-423f-9433-1eb283f1263d" containerName="main" Oct 30 03:04:54.025752 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025660 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b793f8b-7d28-423f-9433-1eb283f1263d" containerName="main" Oct 30 03:04:54.025752 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025673 5035 cpu_manager.go:414] "containerMap: removing container" podUID="fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" containerName="payload-processing" Oct 30 03:04:54.025752 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025678 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" containerName="payload-processing" Oct 30 03:04:54.025752 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025748 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" Oct 30 03:04:54.025942 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025759 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b793f8b-7d28-423f-9433-1eb283f1263d" containerName="main" Oct 30 03:04:54.025942 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025768 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb327c5e-1ce7-4f8e-8bb3-cb6139f66795" containerName="payload-processing" Oct 30 03:04:54.025942 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.025777 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9526682-8fe9-490f-a151-5e220f1eac4e" containerName="revoke-keys" Oct 30 03:04:54.030978 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.030953 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.046143 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.046110 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:04:54.058423 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.058332 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:04:54.058423 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.058398 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:04:54.058423 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.058333 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:04:54.062423 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.062389 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w"] Oct 30 03:04:54.109829 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.109763 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rvt\" (UniqueName: \"kubernetes.io/projected/3c33a3ee-aed9-46b7-a281-6c41507ee969-kube-api-access-n8rvt\") pod \"maas-api-revoke-sub-d576e68b-4cv4w\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.109829 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.109839 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-4cv4w\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.110119 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.109993 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-4cv4w\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.211717 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.211669 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rvt\" (UniqueName: \"kubernetes.io/projected/3c33a3ee-aed9-46b7-a281-6c41507ee969-kube-api-access-n8rvt\") pod \"maas-api-revoke-sub-d576e68b-4cv4w\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.211992 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.211726 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-4cv4w\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.211992 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.211795 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-4cv4w\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.212683 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.212632 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-4cv4w\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.215057 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.215027 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-4cv4w\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.237224 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.237180 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rvt\" (UniqueName: \"kubernetes.io/projected/3c33a3ee-aed9-46b7-a281-6c41507ee969-kube-api-access-n8rvt\") pod \"maas-api-revoke-sub-d576e68b-4cv4w\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.344255 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.344162 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:54.650080 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.650050 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w"] Oct 30 03:04:54.652142 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:04:54.652113 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c33a3ee_aed9_46b7_a281_6c41507ee969.slice/crio-82c91359202ca7f09a1890c395dbf73107f391f94a5a96f2f432702980b75105 WatchSource:0}: Error finding container 82c91359202ca7f09a1890c395dbf73107f391f94a5a96f2f432702980b75105: Status 404 returned error can't find the container with id 82c91359202ca7f09a1890c395dbf73107f391f94a5a96f2f432702980b75105 Oct 30 03:04:54.974015 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:54.973974 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" event={"ID":"3c33a3ee-aed9-46b7-a281-6c41507ee969","Type":"ContainerStarted","Data":"82c91359202ca7f09a1890c395dbf73107f391f94a5a96f2f432702980b75105"} Oct 30 03:04:55.981432 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:55.981389 5035 generic.go:356] "Generic (PLEG): container finished" podID="3c33a3ee-aed9-46b7-a281-6c41507ee969" containerID="34d00dc1ea8e0c2b6a3edbe55ecab2bdfd51625cf2b1988efbf8c36711bec89d" exitCode=0 Oct 30 03:04:55.981890 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:55.981476 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" event={"ID":"3c33a3ee-aed9-46b7-a281-6c41507ee969","Type":"ContainerDied","Data":"34d00dc1ea8e0c2b6a3edbe55ecab2bdfd51625cf2b1988efbf8c36711bec89d"} Oct 30 03:04:57.049554 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.046978 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr"] Oct 30 03:04:57.054661 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.054621 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2bjvr"] Oct 30 03:04:57.132738 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.132707 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:57.242509 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.242277 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c33a3ee-aed9-46b7-a281-6c41507ee969-kube-api-access-n8rvt\" (UniqueName: \"kubernetes.io/projected/3c33a3ee-aed9-46b7-a281-6c41507ee969-kube-api-access-n8rvt\") pod \"3c33a3ee-aed9-46b7-a281-6c41507ee969\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " Oct 30 03:04:57.242509 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.242385 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-xks-ca\") pod \"3c33a3ee-aed9-46b7-a281-6c41507ee969\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " Oct 30 03:04:57.242509 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.242455 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-openshift-ca\") pod \"3c33a3ee-aed9-46b7-a281-6c41507ee969\" (UID: \"3c33a3ee-aed9-46b7-a281-6c41507ee969\") " Oct 30 03:04:57.243156 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.243092 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-openshift-ca" pod "3c33a3ee-aed9-46b7-a281-6c41507ee969" (UID: "3c33a3ee-aed9-46b7-a281-6c41507ee969"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:04:57.243350 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.243324 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:04:57.246409 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.246367 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-xks-ca" pod "3c33a3ee-aed9-46b7-a281-6c41507ee969" (UID: "3c33a3ee-aed9-46b7-a281-6c41507ee969"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:04:57.246523 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.246393 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c33a3ee-aed9-46b7-a281-6c41507ee969-kube-api-access-n8rvt" pod "3c33a3ee-aed9-46b7-a281-6c41507ee969" (UID: "3c33a3ee-aed9-46b7-a281-6c41507ee969"). InnerVolumeSpecName "kube-api-access-n8rvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:04:57.344522 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.344428 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rvt\" (UniqueName: \"kubernetes.io/projected/3c33a3ee-aed9-46b7-a281-6c41507ee969-kube-api-access-n8rvt\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:04:57.344522 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.344504 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c33a3ee-aed9-46b7-a281-6c41507ee969-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:04:57.642021 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.641933 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136ea7d6-8486-4f60-958a-3752c3655ef2" path="/var/lib/kubelet/pods/136ea7d6-8486-4f60-958a-3752c3655ef2/volumes" Oct 30 03:04:57.995761 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.995723 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" event={"ID":"3c33a3ee-aed9-46b7-a281-6c41507ee969","Type":"ContainerDied","Data":"82c91359202ca7f09a1890c395dbf73107f391f94a5a96f2f432702980b75105"} Oct 30 03:04:57.995970 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.995786 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w" Oct 30 03:04:57.995970 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:04:57.995814 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c91359202ca7f09a1890c395dbf73107f391f94a5a96f2f432702980b75105" Oct 30 03:05:09.134380 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.134327 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns"] Oct 30 03:05:09.135069 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.135045 5035 cpu_manager.go:414] "containerMap: removing container" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" Oct 30 03:05:09.135154 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.135072 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" Oct 30 03:05:09.135154 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.135106 5035 cpu_manager.go:414] "containerMap: removing container" podUID="3c33a3ee-aed9-46b7-a281-6c41507ee969" containerName="revoke-keys" Oct 30 03:05:09.135154 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.135115 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c33a3ee-aed9-46b7-a281-6c41507ee969" containerName="revoke-keys" Oct 30 03:05:09.135351 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.135260 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c33a3ee-aed9-46b7-a281-6c41507ee969" containerName="revoke-keys" Oct 30 03:05:09.135351 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.135280 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="63654084-c5cc-4af5-9837-f91d92b3276d" containerName="istio-proxy" Oct 30 03:05:09.139763 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.139728 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.174446 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.174406 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:05:09.179339 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.179291 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns"] Oct 30 03:05:09.180130 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.180101 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:05:09.180279 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.180261 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:05:09.180354 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.180313 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:05:09.201821 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.201767 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf5gm\" (UniqueName: \"kubernetes.io/projected/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-kube-api-access-jf5gm\") pod \"maas-api-revoke-sub-d2bc93bc-t5wns\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.202160 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.202133 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-t5wns\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.202318 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.202300 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-t5wns\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.312556 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.312468 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf5gm\" (UniqueName: \"kubernetes.io/projected/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-kube-api-access-jf5gm\") pod \"maas-api-revoke-sub-d2bc93bc-t5wns\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.339370 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.339329 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-t5wns\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.340817 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.340786 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-t5wns\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.345729 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.345688 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-t5wns\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.357220 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.357181 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-t5wns\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.381323 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.381282 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf5gm\" (UniqueName: \"kubernetes.io/projected/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-kube-api-access-jf5gm\") pod \"maas-api-revoke-sub-d2bc93bc-t5wns\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:09.479702 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:09.475414 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:10.654325 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:10.654278 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns"] Oct 30 03:05:11.166450 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:11.166399 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" event={"ID":"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50","Type":"ContainerStarted","Data":"395551832b9e462271e0250f7f5c68496218e2700d8926db482307a5da7ea02e"} Oct 30 03:05:12.174259 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:12.174160 5035 generic.go:356] "Generic (PLEG): container finished" podID="f15b34be-e7d2-45f4-9cec-b9cf06cf3f50" containerID="57cf3007b052fe87d6650de0f61dbba6ae22afcbe18090e17275cab1deab1c35" exitCode=0 Oct 30 03:05:12.174697 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:12.174240 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" event={"ID":"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50","Type":"ContainerDied","Data":"57cf3007b052fe87d6650de0f61dbba6ae22afcbe18090e17275cab1deab1c35"} Oct 30 03:05:13.326794 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.326759 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:13.377617 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.377580 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-kube-api-access-jf5gm\" (UniqueName: \"kubernetes.io/projected/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-kube-api-access-jf5gm\") pod \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " Oct 30 03:05:13.377841 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.377709 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-xks-ca\") pod \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " Oct 30 03:05:13.377841 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.377736 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-openshift-ca\") pod \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\" (UID: \"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50\") " Oct 30 03:05:13.378212 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.378182 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-openshift-ca" pod "f15b34be-e7d2-45f4-9cec-b9cf06cf3f50" (UID: "f15b34be-e7d2-45f4-9cec-b9cf06cf3f50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:05:13.379997 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.379963 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-kube-api-access-jf5gm" pod "f15b34be-e7d2-45f4-9cec-b9cf06cf3f50" (UID: "f15b34be-e7d2-45f4-9cec-b9cf06cf3f50"). InnerVolumeSpecName "kube-api-access-jf5gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:05:13.380113 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.379992 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-xks-ca" pod "f15b34be-e7d2-45f4-9cec-b9cf06cf3f50" (UID: "f15b34be-e7d2-45f4-9cec-b9cf06cf3f50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:05:13.479010 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.478964 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf5gm\" (UniqueName: \"kubernetes.io/projected/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-kube-api-access-jf5gm\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:05:13.479010 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.479006 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-xks-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:05:13.479010 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:13.479016 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50-maas-api-openshift-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:05:14.193063 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:14.192985 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" event={"ID":"f15b34be-e7d2-45f4-9cec-b9cf06cf3f50","Type":"ContainerDied","Data":"395551832b9e462271e0250f7f5c68496218e2700d8926db482307a5da7ea02e"} Oct 30 03:05:14.193063 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:14.193073 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns" Oct 30 03:05:14.193553 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:14.193102 5035 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="395551832b9e462271e0250f7f5c68496218e2700d8926db482307a5da7ea02e" Oct 30 03:05:17.628959 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:05:17.628915 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:05:21.359904 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.359806 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k"] Oct 30 03:05:21.361489 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.361411 5035 cpu_manager.go:414] "containerMap: removing container" podUID="f15b34be-e7d2-45f4-9cec-b9cf06cf3f50" containerName="revoke-keys" Oct 30 03:05:21.361489 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.361492 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15b34be-e7d2-45f4-9cec-b9cf06cf3f50" containerName="revoke-keys" Oct 30 03:05:21.361909 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.361863 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15b34be-e7d2-45f4-9cec-b9cf06cf3f50" containerName="revoke-keys" Oct 30 03:05:21.368072 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.367996 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.393928 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.393814 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k"] Oct 30 03:05:21.394880 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.394834 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:05:21.395195 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.394897 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:05:21.397684 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.397236 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 03:05:21.397684 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.397394 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xzt6k\"" Oct 30 03:05:21.397684 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.397487 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 03:05:21.484150 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.483971 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-maas-api-tls\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.484877 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.484352 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-metrics-certs\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.484877 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.484671 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-openshift-service-ca\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.484877 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.484780 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxv7r\" (UniqueName: \"kubernetes.io/projected/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-kube-api-access-nxv7r\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.586733 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.586683 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-maas-api-tls\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.586961 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.586765 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-metrics-certs\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.586961 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.586811 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-openshift-service-ca\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.586961 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.586843 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxv7r\" (UniqueName: \"kubernetes.io/projected/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-kube-api-access-nxv7r\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.587595 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.587570 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-openshift-service-ca\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.589348 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.589317 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-maas-api-tls\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.589510 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.589390 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-metrics-certs\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.611747 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.611663 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxv7r\" (UniqueName: \"kubernetes.io/projected/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-kube-api-access-nxv7r\") pod \"maas-api-647d7b7446-k2v5k\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:21.708718 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:21.708674 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:22.030794 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:22.030758 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k"] Oct 30 03:05:22.032986 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:05:22.032952 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038dcefc_7f8f_4539_8bc3_c4e0453ab4cf.slice/crio-52b5c1c137f6311c2d6f17d9cb6fb7ffcc5b9f97b49160e6504c49b022f08c79 WatchSource:0}: Error finding container 52b5c1c137f6311c2d6f17d9cb6fb7ffcc5b9f97b49160e6504c49b022f08c79: Status 404 returned error can't find the container with id 52b5c1c137f6311c2d6f17d9cb6fb7ffcc5b9f97b49160e6504c49b022f08c79 Oct 30 03:05:22.034399 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:22.034372 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:05:22.248062 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:22.247980 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" event={"ID":"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf","Type":"ContainerStarted","Data":"52b5c1c137f6311c2d6f17d9cb6fb7ffcc5b9f97b49160e6504c49b022f08c79"} Oct 30 03:05:23.255663 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:23.255562 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" event={"ID":"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf","Type":"ContainerStarted","Data":"54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c"} Oct 30 03:05:23.256166 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:23.255663 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:23.300968 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:23.300896 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" podStartSLOduration=2.012401065 podStartE2EDuration="2.300873795s" totalImagesPullingTime="288.47273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:23.298083637 +0000 UTC m=+5801.801015858" watchObservedRunningTime="2026-10-30 03:05:23.300873795 +0000 UTC m=+5801.803806016" Oct 30 03:05:28.571774 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.571729 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k"] Oct 30 03:05:28.572285 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.571930 5035 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" podUID="038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" containerName="maas-api" containerID="cri-o://54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c" gracePeriod=30 Oct 30 03:05:28.578305 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.578261 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:28.617951 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.617912 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl"] Oct 30 03:05:28.622491 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.622460 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.646563 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.646509 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl"] Oct 30 03:05:28.767094 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.767048 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ca759-2e5e-4fd9-b7fc-86415774c463-metrics-certs\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.767321 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.767104 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpsms\" (UniqueName: \"kubernetes.io/projected/9a4ca759-2e5e-4fd9-b7fc-86415774c463-kube-api-access-dpsms\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.767321 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.767217 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4ca759-2e5e-4fd9-b7fc-86415774c463-openshift-service-ca\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.767321 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.767308 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4ca759-2e5e-4fd9-b7fc-86415774c463-maas-api-tls\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.836751 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.836676 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:28.868275 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.868233 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4ca759-2e5e-4fd9-b7fc-86415774c463-maas-api-tls\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.868521 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.868333 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ca759-2e5e-4fd9-b7fc-86415774c463-metrics-certs\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.868521 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.868353 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsms\" (UniqueName: \"kubernetes.io/projected/9a4ca759-2e5e-4fd9-b7fc-86415774c463-kube-api-access-dpsms\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.868521 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.868382 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4ca759-2e5e-4fd9-b7fc-86415774c463-openshift-service-ca\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.869052 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.869026 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4ca759-2e5e-4fd9-b7fc-86415774c463-openshift-service-ca\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.870887 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.870859 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ca759-2e5e-4fd9-b7fc-86415774c463-metrics-certs\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.871111 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.871086 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4ca759-2e5e-4fd9-b7fc-86415774c463-maas-api-tls\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.894949 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.894902 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpsms\" (UniqueName: \"kubernetes.io/projected/9a4ca759-2e5e-4fd9-b7fc-86415774c463-kube-api-access-dpsms\") pod \"maas-api-6cf77bf7d-8f6xl\" (UID: \"9a4ca759-2e5e-4fd9-b7fc-86415774c463\") " pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.966079 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.966026 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:28.969672 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.969362 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-openshift-service-ca\") pod \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " Oct 30 03:05:28.969672 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.969483 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-metrics-certs\") pod \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " Oct 30 03:05:28.969672 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.969612 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-kube-api-access-nxv7r\" (UniqueName: \"kubernetes.io/projected/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-kube-api-access-nxv7r\") pod \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " Oct 30 03:05:28.969672 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.969652 5035 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-maas-api-tls\") pod \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\" (UID: \"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf\") " Oct 30 03:05:28.969923 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.969785 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-openshift-service-ca" pod "038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" (UID: "038dcefc-7f8f-4539-8bc3-c4e0453ab4cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:05:28.969981 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.969968 5035 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-openshift-service-ca\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:05:28.971734 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.971699 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-metrics-certs" pod "038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" (UID: "038dcefc-7f8f-4539-8bc3-c4e0453ab4cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:05:28.971866 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.971749 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-maas-api-tls" pod "038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" (UID: "038dcefc-7f8f-4539-8bc3-c4e0453ab4cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:05:28.972282 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:28.972261 5035 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-kube-api-access-nxv7r" pod "038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" (UID: "038dcefc-7f8f-4539-8bc3-c4e0453ab4cf"). InnerVolumeSpecName "kube-api-access-nxv7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:05:29.070594 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.070548 5035 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-metrics-certs\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:05:29.070594 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.070590 5035 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxv7r\" (UniqueName: \"kubernetes.io/projected/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-kube-api-access-nxv7r\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:05:29.070594 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.070600 5035 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf-maas-api-tls\") on node \"ip-10-0-142-110.ec2.internal\" DevicePath \"\"" Oct 30 03:05:29.275950 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.275900 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl"] Oct 30 03:05:29.278194 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:05:29.278167 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4ca759_2e5e_4fd9_b7fc_86415774c463.slice/crio-33fc72f0547fe94cb1310a3d2d40a8de38aae0b0e28b3326500ca374152dc7ff WatchSource:0}: Error finding container 33fc72f0547fe94cb1310a3d2d40a8de38aae0b0e28b3326500ca374152dc7ff: Status 404 returned error can't find the container with id 33fc72f0547fe94cb1310a3d2d40a8de38aae0b0e28b3326500ca374152dc7ff Oct 30 03:05:29.298786 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.298750 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" event={"ID":"9a4ca759-2e5e-4fd9-b7fc-86415774c463","Type":"ContainerStarted","Data":"33fc72f0547fe94cb1310a3d2d40a8de38aae0b0e28b3326500ca374152dc7ff"} Oct 30 03:05:29.300143 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.300110 5035 generic.go:356] "Generic (PLEG): container finished" podID="038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" containerID="54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c" exitCode=0 Oct 30 03:05:29.300289 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.300192 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" event={"ID":"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf","Type":"ContainerDied","Data":"54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c"} Oct 30 03:05:29.300289 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.300218 5035 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" Oct 30 03:05:29.300289 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.300251 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k" event={"ID":"038dcefc-7f8f-4539-8bc3-c4e0453ab4cf","Type":"ContainerDied","Data":"52b5c1c137f6311c2d6f17d9cb6fb7ffcc5b9f97b49160e6504c49b022f08c79"} Oct 30 03:05:29.300289 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.300275 5035 scope.go:122] "RemoveContainer" containerID="54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c" Oct 30 03:05:29.310849 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.310818 5035 scope.go:122] "RemoveContainer" containerID="54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c" Oct 30 03:05:29.311230 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:05:29.311202 5035 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c\": container with ID starting with 54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c not found: ID does not exist" containerID="54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c" Oct 30 03:05:29.311320 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.311243 5035 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c"} err="failed to get container status \"54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c\": rpc error: code = NotFound desc = could not find container \"54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c\": container with ID starting with 54601986a63c4591318727c607e613a89d05340daa5e059d5b583dd123bbd61c not found: ID does not exist" Oct 30 03:05:29.352244 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.352197 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k"] Oct 30 03:05:29.365023 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.364984 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-647d7b7446-k2v5k"] Oct 30 03:05:29.646878 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:29.646849 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" path="/var/lib/kubelet/pods/038dcefc-7f8f-4539-8bc3-c4e0453ab4cf/volumes" Oct 30 03:05:30.310660 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:30.310621 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" event={"ID":"9a4ca759-2e5e-4fd9-b7fc-86415774c463","Type":"ContainerStarted","Data":"9942ef41023986cddab449fcb5033ebafafb5a7351e14b7248381649071bfca1"} Oct 30 03:05:30.310880 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:30.310747 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:30.354098 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:30.354040 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" podStartSLOduration=2.009891114 podStartE2EDuration="2.35402597s" totalImagesPullingTime="344.134856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:30.353498662 +0000 UTC m=+5808.856430883" watchObservedRunningTime="2026-10-30 03:05:30.35402597 +0000 UTC m=+5808.856958221" Oct 30 03:05:36.327611 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:36.327568 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cf77bf7d-8f6xl" Oct 30 03:05:45.774485 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:45.774449 5035 scope.go:122] "RemoveContainer" containerID="55feda34144455de99f431a3e3302b4dfa91fcff19061d6514a026f5dc813724" Oct 30 03:05:45.783961 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:45.783933 5035 scope.go:122] "RemoveContainer" containerID="27f83057ae0b9be986dad7005a2ea8a52a872e2a005823eba1a6feb4783b1947" Oct 30 03:05:45.792963 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:45.792936 5035 scope.go:122] "RemoveContainer" containerID="be7793f5ddad31f158ec8a524a5d2b2f38b031f6b8d5545f416af03565c47add" Oct 30 03:05:45.802120 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:05:45.802096 5035 scope.go:122] "RemoveContainer" containerID="ca2268096da3e953f83310d329659397a7562cc8ab00d02eccf4384b2d20e8e2" Oct 30 03:05:47.628112 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:05:47.628070 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:06:09.089554 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:09.088596 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9"] Oct 30 03:06:09.113556 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:09.107693 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9h4p9"] Oct 30 03:06:09.657956 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:09.657909 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19efc19a-a22c-4560-b974-bb2a1e3bfb79" path="/var/lib/kubelet/pods/19efc19a-a22c-4560-b974-bb2a1e3bfb79/volumes" Oct 30 03:06:17.629405 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:06:17.629365 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:06:21.032376 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:21.032336 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm"] Oct 30 03:06:21.045887 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:21.045838 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-84mpm"] Oct 30 03:06:21.647998 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:21.647889 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1cda16d-f399-4592-abac-88d065bac440" path="/var/lib/kubelet/pods/d1cda16d-f399-4592-abac-88d065bac440/volumes" Oct 30 03:06:31.490555 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:31.489744 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:06:31.490555 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:31.489881 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:06:31.551553 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:31.548061 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:06:31.551553 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:31.548159 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:06:45.846477 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:45.846435 5035 scope.go:122] "RemoveContainer" containerID="01ec9740686f21be2f1bd5c8395f07d2a9ceff0fc87d689422fd779155e35176" Oct 30 03:06:45.856244 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:06:45.856220 5035 scope.go:122] "RemoveContainer" containerID="02d78aa719c697edd38683bedfce5567e2931c2f36797b295b97c01cc6738bf0" Oct 30 03:06:47.631556 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:06:47.631502 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:07:17.641419 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:07:17.641374 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:07:44.034001 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:07:44.033954 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4"] Oct 30 03:07:44.047810 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:07:44.047770 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9727a5d-jvbw4"] Oct 30 03:07:45.642104 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:07:45.642068 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e626c0b2-7c2a-45ab-99d2-9463524e4408" path="/var/lib/kubelet/pods/e626c0b2-7c2a-45ab-99d2-9463524e4408/volumes" Oct 30 03:07:45.885025 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:07:45.884985 5035 scope.go:122] "RemoveContainer" containerID="64f304886e6fa566caf42322cef02b89f2ab7e8b6ef2d4e2cab64d3de9f58322" Oct 30 03:07:45.894510 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:07:45.894489 5035 scope.go:122] "RemoveContainer" containerID="bc5f69b182dfdcd253dd053063c494d49549468b07528a2d8f5e870f1aaeb9d9" Oct 30 03:07:45.903460 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:07:45.903437 5035 scope.go:122] "RemoveContainer" containerID="d1465217979a5c946ff96d3d53f66918ef42c6d576b4e52ac3cf96b6a1976802" Oct 30 03:07:47.630858 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:07:47.630812 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:08:01.495633 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:01.494750 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zvgvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:08:01.495633 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:01.495029 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zvgvw" podUID="a71091cb-5eac-44d9-8cba-bbc3f04d66d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:08:01.539652 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:01.538818 5035 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4spcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:08:01.539652 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:01.539090 5035 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4spcb" podUID="807bae01-b730-4075-b8ec-eed162d6afcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:08:13.032272 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:13.032225 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh"] Oct 30 03:08:13.044046 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:13.044005 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wtrvh"] Oct 30 03:08:13.642241 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:13.642191 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9526682-8fe9-490f-a151-5e220f1eac4e" path="/var/lib/kubelet/pods/d9526682-8fe9-490f-a151-5e220f1eac4e/volumes" Oct 30 03:08:17.627999 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:08:17.627959 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:08:44.343985 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:44.343952 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 03:08:44.356941 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:44.356903 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 03:08:44.362344 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:44.362308 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 03:08:44.438862 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:44.438827 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 03:08:44.451727 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:44.451697 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5kdp7_bd24e16c-c31b-4af0-b518-cfe8c85937ac/cluster-samples-operator/0.log" Oct 30 03:08:44.456433 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:44.456407 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 03:08:45.935649 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:08:45.935602 5035 scope.go:122] "RemoveContainer" containerID="609b5582b3e06d2db590e2ef3a38b367680597be83c5356a1df2c5e8f61041bc" Oct 30 03:08:47.764905 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:08:47.764854 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:09:17.864553 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:09:17.863953 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:09:46.207544 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:09:46.207488 5035 scope.go:122] "RemoveContainer" containerID="1828069aa4fde22e5ca869ab66f4f9ccb2adf411cf5d63ded6e1d9c1da3d6184" Oct 30 03:09:47.653157 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:09:47.653116 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:09:58.036882 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:09:58.036764 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w"] Oct 30 03:09:58.040399 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:09:58.040358 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4cv4w"] Oct 30 03:09:59.692597 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:09:59.692504 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c33a3ee-aed9-46b7-a281-6c41507ee969" path="/var/lib/kubelet/pods/3c33a3ee-aed9-46b7-a281-6c41507ee969/volumes" Oct 30 03:10:14.032294 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:14.032249 5035 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns"] Oct 30 03:10:14.045359 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:14.045323 5035 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-t5wns"] Oct 30 03:10:15.641954 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:15.641914 5035 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15b34be-e7d2-45f4-9cec-b9cf06cf3f50" path="/var/lib/kubelet/pods/f15b34be-e7d2-45f4-9cec-b9cf06cf3f50/volumes" Oct 30 03:10:17.629118 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:10:17.629072 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:10:43.965897 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:43.965858 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6cf77bf7d-8f6xl_9a4ca759-2e5e-4fd9-b7fc-86415774c463/maas-api/0.log" Oct 30 03:10:47.629025 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:10:47.628983 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:10:50.153145 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:50.153021 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-45k2d_e33eacd4-92b9-47b7-b441-7f0b47bd6071/manager/0.log" Oct 30 03:10:50.574955 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:50.574918 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8fhqf_6e497615-20c6-4568-a41b-f8220a1f2e6c/kuadrant-console-plugin/0.log" Oct 30 03:10:51.333539 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:51.333469 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-57dvj_adaf904e-ceb8-48d6-816f-3316258fe072/manager/0.log" Oct 30 03:10:52.386435 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:52.386396 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv_a3315f88-afe8-4cf3-9f2b-16567421aae8/istio-proxy/0.log" Oct 30 03:10:53.392101 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:53.392057 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-4spcb_807bae01-b730-4075-b8ec-eed162d6afcd/istio-proxy/0.log" Oct 30 03:10:54.026213 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:54.026159 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-g55b2_4dae7370-d6fd-4ad6-bcc3-2bfb55be9904/istio-proxy/0.log" Oct 30 03:10:54.849596 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:54.849548 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-rt6p2_27d77144-491e-48c4-80c2-530ab45394cd/istio-proxy/0.log" Oct 30 03:10:55.078563 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:55.078505 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-zvgvw_a71091cb-5eac-44d9-8cba-bbc3f04d66d6/istio-proxy/0.log" Oct 30 03:10:55.510373 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:55.510339 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7bf6df7cf-qcpgl_a6b84a83-7da1-4dfa-bb59-2ad03496cc4a/payload-pre-processing/0.log" Oct 30 03:10:55.895018 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:55.894920 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f7cfffd5-4l7bn_02ab5bdf-56b9-4176-af0e-f41d92760b84/router/0.log" Oct 30 03:10:57.341442 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:57.341407 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4dsxw_828bd886-3208-4152-b4bb-551b10a3a254/main/0.log" Oct 30 03:10:57.534807 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:57.534771 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-v5zns_ef8fcd62-a30c-463d-9ffc-a516d8eb7e73/main/0.log" Oct 30 03:10:57.921892 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:57.921852 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/1.log" Oct 30 03:10:58.360131 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:58.360097 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-mk4gj_ed9fd4ef-0aab-454d-b95f-83cbcb4fe367/main/0.log" Oct 30 03:10:58.953905 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:10:58.953869 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dw8g7_dca83999-a27c-4d5b-8ab5-09ccbf01f5a6/main/0.log" Oct 30 03:11:09.614719 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.614509 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-29dhb/must-gather-8zpbk"] Oct 30 03:11:09.616135 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.616028 5035 cpu_manager.go:414] "containerMap: removing container" podUID="038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" containerName="maas-api" Oct 30 03:11:09.616135 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.616084 5035 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" containerName="maas-api" Oct 30 03:11:09.616364 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.616336 5035 memory_manager.go:365] "RemoveStaleState removing state" podUID="038dcefc-7f8f-4539-8bc3-c4e0453ab4cf" containerName="maas-api" Oct 30 03:11:09.622978 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.622877 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-29dhb/must-gather-8zpbk" Oct 30 03:11:09.655147 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.655039 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-29dhb/must-gather-8zpbk"] Oct 30 03:11:09.661268 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.661171 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-29dhb\"/\"kube-root-ca.crt\"" Oct 30 03:11:09.661268 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.661246 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-29dhb\"/\"openshift-service-ca.crt\"" Oct 30 03:11:09.737612 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.737462 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/45126cd8-d109-470a-ad19-ad97249c6267-must-gather-output\") pod \"must-gather-8zpbk\" (UID: \"45126cd8-d109-470a-ad19-ad97249c6267\") " pod="openshift-must-gather-29dhb/must-gather-8zpbk" Oct 30 03:11:09.738450 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.737898 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rr7c\" (UniqueName: \"kubernetes.io/projected/45126cd8-d109-470a-ad19-ad97249c6267-kube-api-access-9rr7c\") pod \"must-gather-8zpbk\" (UID: \"45126cd8-d109-470a-ad19-ad97249c6267\") " pod="openshift-must-gather-29dhb/must-gather-8zpbk" Oct 30 03:11:09.839593 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.839510 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rr7c\" (UniqueName: \"kubernetes.io/projected/45126cd8-d109-470a-ad19-ad97249c6267-kube-api-access-9rr7c\") pod \"must-gather-8zpbk\" (UID: \"45126cd8-d109-470a-ad19-ad97249c6267\") " pod="openshift-must-gather-29dhb/must-gather-8zpbk" Oct 30 03:11:09.839838 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.839642 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/45126cd8-d109-470a-ad19-ad97249c6267-must-gather-output\") pod \"must-gather-8zpbk\" (UID: \"45126cd8-d109-470a-ad19-ad97249c6267\") " pod="openshift-must-gather-29dhb/must-gather-8zpbk" Oct 30 03:11:09.840030 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.840008 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/45126cd8-d109-470a-ad19-ad97249c6267-must-gather-output\") pod \"must-gather-8zpbk\" (UID: \"45126cd8-d109-470a-ad19-ad97249c6267\") " pod="openshift-must-gather-29dhb/must-gather-8zpbk" Oct 30 03:11:09.863791 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.863756 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rr7c\" (UniqueName: \"kubernetes.io/projected/45126cd8-d109-470a-ad19-ad97249c6267-kube-api-access-9rr7c\") pod \"must-gather-8zpbk\" (UID: \"45126cd8-d109-470a-ad19-ad97249c6267\") " pod="openshift-must-gather-29dhb/must-gather-8zpbk" Oct 30 03:11:09.962079 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:09.962030 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-29dhb/must-gather-8zpbk" Oct 30 03:11:10.280484 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:10.280455 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-29dhb/must-gather-8zpbk"] Oct 30 03:11:10.282752 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:11:10.282716 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45126cd8_d109_470a_ad19_ad97249c6267.slice/crio-91a3abbf76eff5a46d48c38a56ae12c29a300e3a1e72851e7086ffe435963a3d WatchSource:0}: Error finding container 91a3abbf76eff5a46d48c38a56ae12c29a300e3a1e72851e7086ffe435963a3d: Status 404 returned error can't find the container with id 91a3abbf76eff5a46d48c38a56ae12c29a300e3a1e72851e7086ffe435963a3d Oct 30 03:11:10.284405 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:10.284385 5035 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:11:10.806837 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:10.806794 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-29dhb/must-gather-8zpbk" event={"ID":"45126cd8-d109-470a-ad19-ad97249c6267","Type":"ContainerStarted","Data":"91a3abbf76eff5a46d48c38a56ae12c29a300e3a1e72851e7086ffe435963a3d"} Oct 30 03:11:15.366706 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:15.366644 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xknfd_a6fed544-0d4f-4a7b-9ba6-1b47bb5eaa92/global-pull-secret-syncer/0.log" Oct 30 03:11:15.523077 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:15.523040 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jlj8q_c36ce70e-acde-4698-89b9-37d85639a669/connection-checker/0.log" Oct 30 03:11:15.636891 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:15.636815 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vg24q_53abc737-fbef-4574-96b8-b9adee4fade2/connection-checker/0.log" Oct 30 03:11:15.753293 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:15.753248 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-z42rk_eadbf8a4-af76-4fab-b804-13e1ac18afcc/connection-checker/0.log" Oct 30 03:11:15.859995 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:15.859950 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-29dhb/must-gather-8zpbk" event={"ID":"45126cd8-d109-470a-ad19-ad97249c6267","Type":"ContainerStarted","Data":"76d2aea400498c8c14ab4d3a451eab901df558571e148ed879966144d21bd9ce"} Oct 30 03:11:15.860210 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:15.860003 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-29dhb/must-gather-8zpbk" event={"ID":"45126cd8-d109-470a-ad19-ad97249c6267","Type":"ContainerStarted","Data":"c19479b96c7ab59c92a0fcee774c82a056b6c566e1d85fa4d88b7fdf636026b7"} Oct 30 03:11:15.907618 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:15.907459 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-29dhb/must-gather-8zpbk" podStartSLOduration=1.8449254929999999 podStartE2EDuration="6.9074382s" totalImagesPullingTime="5.062512707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:15.905581279 +0000 UTC m=+6154.408513500" watchObservedRunningTime="2026-10-30 03:11:15.9074382 +0000 UTC m=+6154.410370423" Oct 30 03:11:16.044849 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:16.044812 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5rdzj_44ef0b6d-c29b-4925-b06f-5136ef9aa906/konnectivity-agent/0.log" Oct 30 03:11:16.514582 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:16.514512 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-110.ec2.internal_ec075332058530a9d0af488e14c4afa0/haproxy/0.log" Oct 30 03:11:17.632935 ip-10-0-142-110 kubenswrapper[5035]: E1030 03:11:17.632896 5035 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hflnn" containerName="extractor" Oct 30 03:11:24.282901 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:24.282790 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-45k2d_e33eacd4-92b9-47b7-b441-7f0b47bd6071/manager/0.log" Oct 30 03:11:24.540587 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:24.540494 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8fhqf_6e497615-20c6-4568-a41b-f8220a1f2e6c/kuadrant-console-plugin/0.log" Oct 30 03:11:25.067341 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:25.067287 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-57dvj_adaf904e-ceb8-48d6-816f-3316258fe072/manager/0.log" Oct 30 03:11:26.258438 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.258381 5035 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq"] Oct 30 03:11:26.266084 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.266047 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.283014 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.282970 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq"] Oct 30 03:11:26.372800 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.372755 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd7bs\"/\"openshift-service-ca.crt\"" Oct 30 03:11:26.373026 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.372808 5035 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd7bs\"/\"kube-root-ca.crt\"" Oct 30 03:11:26.373026 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.372871 5035 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wd7bs\"/\"default-dockercfg-mzxnn\"" Oct 30 03:11:26.460064 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.460019 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-proc\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.460272 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.460120 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-sys\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.460272 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.460192 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxdp\" (UniqueName: \"kubernetes.io/projected/2a65e2da-eb47-4410-81aa-31929bad666e-kube-api-access-qmxdp\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.460272 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.460229 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-lib-modules\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.460416 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.460355 5035 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-podres\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.562514 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.562303 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-proc\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.562514 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.562396 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-sys\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.562514 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.562519 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-proc\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.563244 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.562576 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxdp\" (UniqueName: \"kubernetes.io/projected/2a65e2da-eb47-4410-81aa-31929bad666e-kube-api-access-qmxdp\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.563244 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.562629 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-sys\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.563244 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.562691 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-lib-modules\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.563244 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.562847 5035 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-podres\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.563244 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.562912 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-lib-modules\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.563244 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.563016 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a65e2da-eb47-4410-81aa-31929bad666e-podres\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.589579 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.589509 5035 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxdp\" (UniqueName: \"kubernetes.io/projected/2a65e2da-eb47-4410-81aa-31929bad666e-kube-api-access-qmxdp\") pod \"perf-node-gather-daemonset-glwqq\" (UID: \"2a65e2da-eb47-4410-81aa-31929bad666e\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.881564 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.881264 5035 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:26.983941 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:26.983905 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv_a3315f88-afe8-4cf3-9f2b-16567421aae8/istio-proxy/0.log" Oct 30 03:11:27.213352 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:27.213159 5035 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq"] Oct 30 03:11:27.216650 ip-10-0-142-110 kubenswrapper[5035]: W1030 03:11:27.216614 5035 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2a65e2da_eb47_4410_81aa_31929bad666e.slice/crio-5b52227093a22f9d1b475640feea210e92b99400ec242883f4bbb3f221742e59 WatchSource:0}: Error finding container 5b52227093a22f9d1b475640feea210e92b99400ec242883f4bbb3f221742e59: Status 404 returned error can't find the container with id 5b52227093a22f9d1b475640feea210e92b99400ec242883f4bbb3f221742e59 Oct 30 03:11:27.548031 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:27.547890 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-c4msx_ca1fbc85-1496-4e79-9bc9-330a8540a39e/cluster-monitoring-operator/0.log" Oct 30 03:11:27.954134 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:27.954092 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" event={"ID":"2a65e2da-eb47-4410-81aa-31929bad666e","Type":"ContainerStarted","Data":"4fe4d5449fa0a6fd947b24f7bc68501c32550fc933e3d3c8c9c65e356d5207b6"} Oct 30 03:11:27.954356 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:27.954147 5035 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" event={"ID":"2a65e2da-eb47-4410-81aa-31929bad666e","Type":"ContainerStarted","Data":"5b52227093a22f9d1b475640feea210e92b99400ec242883f4bbb3f221742e59"} Oct 30 03:11:27.954356 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:27.954225 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:27.997379 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:27.997285 5035 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" podStartSLOduration=1.997265975 podStartE2EDuration="1.997265975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:27.996669108 +0000 UTC m=+6166.499601328" watchObservedRunningTime="2026-10-30 03:11:27.997265975 +0000 UTC m=+6166.500198196" Oct 30 03:11:28.348239 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:28.348154 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7qtpf_bac59d27-7d89-4f37-89eb-00d81f30223a/node-exporter/0.log" Oct 30 03:11:28.472403 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:28.472367 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7qtpf_bac59d27-7d89-4f37-89eb-00d81f30223a/kube-rbac-proxy/0.log" Oct 30 03:11:28.594623 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:28.594587 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7qtpf_bac59d27-7d89-4f37-89eb-00d81f30223a/init-textfile/0.log" Oct 30 03:11:29.022504 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:29.022469 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-4spcb_807bae01-b730-4075-b8ec-eed162d6afcd/istio-proxy/0.log" Oct 30 03:11:29.202850 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:29.202815 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-g55b2_4dae7370-d6fd-4ad6-bcc3-2bfb55be9904/istio-proxy/0.log" Oct 30 03:11:29.440173 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:29.440081 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-rt6p2_27d77144-491e-48c4-80c2-530ab45394cd/istio-proxy/0.log" Oct 30 03:11:29.499942 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:29.499907 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-zvgvw_a71091cb-5eac-44d9-8cba-bbc3f04d66d6/istio-proxy/0.log" Oct 30 03:11:29.620785 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:29.620744 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7bf6df7cf-qcpgl_a6b84a83-7da1-4dfa-bb59-2ad03496cc4a/payload-pre-processing/0.log" Oct 30 03:11:29.745350 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:29.745313 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f7cfffd5-4l7bn_02ab5bdf-56b9-4176-af0e-f41d92760b84/router/0.log" Oct 30 03:11:30.636420 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:30.636388 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ldmgm_a0904009-0a90-40f2-b917-fecd405d7ec3/prometheus-operator/0.log" Oct 30 03:11:30.745720 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:30.745684 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ldmgm_a0904009-0a90-40f2-b917-fecd405d7ec3/kube-rbac-proxy/0.log" Oct 30 03:11:30.865523 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:30.865488 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-twz6h_eb6ab779-82b5-4347-8847-1ebad5abe3ea/prometheus-operator-admission-webhook/0.log" Oct 30 03:11:33.501985 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:33.501893 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-6sdlf_bc3b9845-9528-42c3-bdae-dc73fec82643/networking-console-plugin/0.log" Oct 30 03:11:33.636787 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:33.636749 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv_a3315f88-afe8-4cf3-9f2b-16567421aae8/istio-proxy/0.log" Oct 30 03:11:33.974950 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:33.974915 5035 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-glwqq" Oct 30 03:11:34.068587 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:34.068547 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-45k2d_e33eacd4-92b9-47b7-b441-7f0b47bd6071/manager/0.log" Oct 30 03:11:34.189150 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:34.189118 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8fhqf_6e497615-20c6-4568-a41b-f8220a1f2e6c/kuadrant-console-plugin/0.log" Oct 30 03:11:34.663418 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:34.663381 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-57dvj_adaf904e-ceb8-48d6-816f-3316258fe072/manager/0.log" Oct 30 03:11:34.970927 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:34.970894 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/3.log" Oct 30 03:11:34.979093 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:34.979059 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rzw29_7c481c3b-feae-4510-83c5-0dc08ce51db0/console-operator/4.log" Oct 30 03:11:35.647387 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:35.647350 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-4spcb_807bae01-b730-4075-b8ec-eed162d6afcd/istio-proxy/0.log" Oct 30 03:11:35.839109 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:35.839074 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-g55b2_4dae7370-d6fd-4ad6-bcc3-2bfb55be9904/istio-proxy/0.log" Oct 30 03:11:36.085170 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:36.085112 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-rt6p2_27d77144-491e-48c4-80c2-530ab45394cd/istio-proxy/0.log" Oct 30 03:11:36.148096 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:36.148061 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-zvgvw_a71091cb-5eac-44d9-8cba-bbc3f04d66d6/istio-proxy/0.log" Oct 30 03:11:36.270670 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:36.270633 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7bf6df7cf-qcpgl_a6b84a83-7da1-4dfa-bb59-2ad03496cc4a/payload-pre-processing/0.log" Oct 30 03:11:36.401256 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:36.401151 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f7cfffd5-4l7bn_02ab5bdf-56b9-4176-af0e-f41d92760b84/router/0.log" Oct 30 03:11:37.725308 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:37.725257 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-srr2t_2986b483-9c1e-4d59-89d5-01e287807f8a/volume-data-source-validator/0.log" Oct 30 03:11:40.863125 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:40.862907 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dvvl4_aad411a5-bb79-4e0c-abd3-85c5ba535c93/dns/0.log" Oct 30 03:11:40.977500 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:40.977434 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dvvl4_aad411a5-bb79-4e0c-abd3-85c5ba535c93/kube-rbac-proxy/0.log" Oct 30 03:11:41.122185 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:41.122075 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv_a3315f88-afe8-4cf3-9f2b-16567421aae8/istio-proxy/0.log" Oct 30 03:11:41.329419 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:41.329386 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bv8wk_22ff8c4e-b07d-417b-b4d3-61560b014b77/dns-node-resolver/0.log" Oct 30 03:11:41.391026 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:41.390933 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-45k2d_e33eacd4-92b9-47b7-b441-7f0b47bd6071/manager/0.log" Oct 30 03:11:41.499304 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:41.499274 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8fhqf_6e497615-20c6-4568-a41b-f8220a1f2e6c/kuadrant-console-plugin/0.log" Oct 30 03:11:41.924443 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:41.924375 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-57dvj_adaf904e-ceb8-48d6-816f-3316258fe072/manager/0.log" Oct 30 03:11:42.800837 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:42.800782 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-99d5968b4-7dzgs_087e4fbf-a966-4c9f-998c-e55b909a7ea8/registry/0.log" Oct 30 03:11:42.902792 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:42.902751 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8kspl_acf8cbe6-582e-408b-85a9-d969e18b4682/node-ca/0.log" Oct 30 03:11:43.002643 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:43.002488 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-4spcb_807bae01-b730-4075-b8ec-eed162d6afcd/istio-proxy/0.log" Oct 30 03:11:43.190798 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:43.190676 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-g55b2_4dae7370-d6fd-4ad6-bcc3-2bfb55be9904/istio-proxy/0.log" Oct 30 03:11:43.434909 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:43.434866 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-rt6p2_27d77144-491e-48c4-80c2-530ab45394cd/istio-proxy/0.log" Oct 30 03:11:43.495681 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:43.495644 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-zvgvw_a71091cb-5eac-44d9-8cba-bbc3f04d66d6/istio-proxy/0.log" Oct 30 03:11:43.612807 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:43.612744 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7bf6df7cf-qcpgl_a6b84a83-7da1-4dfa-bb59-2ad03496cc4a/payload-pre-processing/0.log" Oct 30 03:11:43.744035 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:43.743796 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f7cfffd5-4l7bn_02ab5bdf-56b9-4176-af0e-f41d92760b84/router/0.log" Oct 30 03:11:45.188316 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:45.188281 5035 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7c5mcwv_a3315f88-afe8-4cf3-9f2b-16567421aae8/istio-proxy/0.log" Oct 30 03:11:46.242735 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:46.242647 5035 scope.go:122] "RemoveContainer" containerID="34d00dc1ea8e0c2b6a3edbe55ecab2bdfd51625cf2b1988efbf8c36711bec89d" Oct 30 03:11:46.256360 ip-10-0-142-110 kubenswrapper[5035]: I1030 03:11:46.256333 5035 scope.go:122] "RemoveContainer" containerID="57cf3007b052fe87d6650de0f61dbba6ae22afcbe18090e17275cab1deab1c35"